Caching and updating of dense 3D reconstruction data

ABSTRACT

A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Altitude information, which may be expressed as a building floor for indoor environments, may be associated with sparse and/or dense representations of the physical world, to increase the accuracy of localization results and/or in rendering virtual content more realistically.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application Ser. No. 62/867,108, filed on Jun. 26, 2019 andentitled “CACHING AND UPDATING OF DENSE 3D RECONSTRUCTION DATA,” whichis hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

This application relates generally to cross reality systems that use a3D world reconstruction to render scenes.

BACKGROUND

Computers may control human user interfaces to create an X Reality (XRor cross reality) environment in which some or all of the XRenvironment, as perceived by the user, is generated by the computer.These XR environments may be virtual reality (VR), augmented reality(AR), and mixed reality (MR) environments, in which some or all of an XRenvironment may be generated by computers using, in part, data thatdescribes the environment. This data may describe, for example, virtualobjects that may be rendered in a way that users sense or perceive as apart of a physical world and can interact with the virtual objects. Theuser may experience these virtual objects as a result of the data beingrendered and presented through a user interface device, such as, forexample, a head-mounted display device. The data may be displayed to theuser to see, or may control audio that is played for the user to hear,or may control a tactile (or haptic) interface, enabling the user toexperience touch sensations that the user senses or perceives as feelingthe virtual object.

XR systems may be useful for many applications, spanning the fields ofscientific visualization, medical training, engineering design andprototyping, tele-manipulation and tele-presence, and personalentertainment. AR and MR, in contrast to VR, include one or more virtualobjects in relation to real objects of the physical world. Theexperience of virtual objects interacting with real objects greatlyenhances the user's enjoyment in using the XR system, and also opens thedoor for a variety of applications that present realistic and readilyunderstandable information about how the physical world might bealtered.

BRIEF SUMMARY

Aspects of the present application relate to methods and apparatus forcaching and updating 3D reconstruction data. The inventors haverecognized and appreciated techniques to cache and update dense 3Dreconstruction data in real-time on devices with limited computationalresource, such as mobile devices. These techniques may be used together,separately, or in any suitable combination.

Some embodiments relate to a portable electronic system. The portableelectronic system comprises at least one sensor configured to capturethree-dimensional (3D) information about objects in a physical world;local memory storing computer executable instructions; a transceiverconfigured for communication over a computer network with remote memory;and a processor communicatively coupled to the at least one sensor, thelocal memory, and the transceiver, wherein the processor executes thecomputer executable instructions to provide a 3D representation of aportion of the physical world based at least in part on the 3Dinformation about the objects in the physical world. The 3Drepresentation of the portion of the physical world comprises aplurality of blocks, each block including values representing objects ina region of portion of the physical world at a corresponding point intime and indicating a floor. The computer executable instructionscomprise instructions for identifying a subset of the plurality ofblocks corresponding to incoming sensor-captured information.

In some embodiments, the values indicating the floors of the blocks aredetermined based at least in part on the 3D information about objects inthe physical world.

In some embodiments, the at least one sensor comprises image sensors.

In some embodiments, at least a portion of the at least one sensor isconfigured to provide at least a portion of the incoming sensor-capturedinformation that a location of the portable electronic system isdetermined based at least in part on.

In some embodiments, the location of the portable electronic system isof varying degrees of granularity.

In some embodiments, the location of the portable electronic systemcomprises the values indicating the floors of corresponding blocks.

In some embodiments, the portable electronic system comprises one ormore of magnetometers, altimeters, and GPS sensors configured to provideat least a portion of the incoming sensor-captured information.

In some embodiments, the values indicating the floors of the blocks aredetermined based at least in part on data captured by the one or more ofmagnetometers, altimeters, and GPS sensors.

In some embodiments, a location of the portable electronic system isdetermined based at least in part on data captured by the one or more ofmagnetometers, altimeters, and GPS sensors.

In some embodiments, wherein the location of the portable electronicsystem is of varying degrees of granularity.

Some embodiments relate to a method of operating a portable electronicsystem. The method comprises with at least one sensor, capturing sensorinformation about a physical world; and with at least one processor:processing the sensor information to: produce a representation of aportion of the physical world proximate the portable electronic device;and associate altitude information with the portion of the physicalworld.

In some embodiments, the altitude information comprises an identifier ofa floor of a building.

In some embodiments, associating altitude information with the portionof the physical world comprises processing an output of a visual sensorto detect an indicator of the floor.

In some embodiments, associating altitude information with the portionof the physical world comprises tracking changes in floor.

In some embodiments, tracking changes in floor comprises processingvisual images to detect traversal of a stairway or escalator.

In some embodiments, the representation of the portion of the physicalworld is a sparse representation.

In some embodiments, the method comprises sending a localization request(a) based on the representation of the portion of the physical world;and (b) including the altitude information.

In some embodiments, the representation of the portion of the physicalworld is a first portion of a dense representation.

In some embodiments, the dense representation comprises a plurality ofportions, including the first portion, each portion comprisingassociated altitude information. The method further comprises performingocclusion processing based on the plurality of portions, the occlusionprocessing comprising ordering surface information in portions of theplurality of portions based on the altitude information associated withthe respective portions.

Some embodiments relate to a non-transitory computer-readable mediumwith instructions stored thereon, that when executed on a processor,perform the acts comprising: processing sensor information about aphysical world to: produce a representation of a portion of the physicalworld proximate a portable electronic device; and associate altitudeinformation with the portion of the physical world.

The foregoing summary is provided by way of illustration and is notintended to be limiting.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a sketch illustrating an example of a simplified augmentedreality (AR) scene, according to some embodiments.

FIG. 2 is a sketch of an exemplary simplified AR scene, showingexemplary World Reconstruction use cases including visual occlusion,physics-based interactions, and environment reasoning, according to someembodiments.

FIG. 3 is a schematic diagram illustrating data flow in an AR systemconfigured to provide an experience of AR contents interacting with aphysical world, according to some embodiments.

FIG. 4 is a schematic diagram illustrating an example of an AR displaysystem, according to some embodiments.

FIG. 5A is a schematic diagram illustrating a user wearing an AR displaysystem rendering AR content as the user moves through a physical worldenvironment, according to some embodiments.

FIG. 5B is a schematic diagram illustrating a viewing optics assemblyand attendant components, according to some embodiments.

FIG. 6 is a schematic diagram illustrating an AR system using a worldreconstruction system, according to some embodiments.

FIG. 7A is a schematic diagram illustrating a 3D space discretized intovoxels, according to some embodiments.

FIG. 7B is a schematic diagram illustrating a reconstruction range withrespect to a single viewpoint, according to some embodiments.

FIG. 7C is a schematic diagram illustrating a perception range withrespect to a reconstruction range at a single position, according tosome embodiments.

FIGS. 8A-F are schematic diagrams illustrating reconstructing a surfacein a physical world into a voxel model by an image sensor viewing thesurface from multiple positions and viewpoints, according to someembodiments.

FIG. 9 is a schematic diagram illustrating a scene represented by brickscomprising voxels, a surface in the scene, and a depth sensor capturingthe surface in a depth image, according to some embodiments.

FIG. 10A is a schematic diagram illustrating a 3D space represented byeight bricks.

FIG. 10B is a schematic diagram illustrating a voxel grid in a brick ofFIG. 10A.

FIG. 11 is a schematic diagram illustrating a volumetric representationhierarchy, according to some embodiments.

FIG. 12 is a flow chart, illustrating a method of operating a computingsystem to generate a 3D reconstruction of a scene, according to someembodiments.

FIG. 13 is a flow chart, illustrating a method of culling a portion ofthe plurality of bricks against a camera frustum of the depth sensor inFIG. 12, according to some embodiments.

FIG. 14 is a flow chart, illustrating a method of conducting a camerafrustum acceptance test in FIG. 13, according to some embodiments.

FIG. 15 is a flow chart, illustrating a method of culling a portion ofthe first plurality of bricks against the depth image in FIG. 12,according to some embodiments.

FIG. 16 is a flow chart, illustrating a method of conducting a firstdepth image acceptance test in FIG. 15, according to some embodiments.

FIG. 17 is a flow chart, illustrating a method of conducting a seconddepth image acceptance test in FIG. 15, according to some embodiments.

FIG. 18 shows a table that is used by a method of categorizing all thepixels in the rectangular with respect to a minimum brick value (bmin)and a maximum brick value (bmax) in FIG. 17, according to someembodiments.

FIGS. 19A-F are schematic diagrams illustrating culling bricks against acamera frustum, according to some embodiments.

FIGS. 20A-B are schematic diagrams illustrating culling bricks against adepth image including a surface, according to some embodiments.

FIG. 21 is a schematic diagram illustrating a plane extraction system,according to some embodiments.

FIG. 22 is a schematic diagram illustrating portions of the planeextraction system of FIG. 21 with details on Plane Extraction, accordingto some embodiments.

FIG. 23 is a schematic diagram illustrating a scene represented bybricks comprising voxels, and exemplary plane data in the scene,according to some embodiments.

FIG. 24 is a schematic diagram illustrating Plane Data Store of FIG. 21,according to some embodiments.

FIG. 25 is a schematic diagram illustrating Planar Geometry Extractionwhen a Plane Query is sent to a Plane Data Store of FIG. 21, accordingto some embodiments.

FIG. 26A is a schematic diagram illustrating generating Plane CoveragePoints of FIG. 25, according to some embodiments.

FIG. 26B is a schematic diagram illustrating various exemplary planargeometry representations, which may be extracted from an exemplaryrasterized plane mask, according to some embodiments.

FIG. 27 shows a mesh for a scene, according to some embodiments.

FIG. 28A shows the scene of FIG. 27 represented by outer rectangularplanes, according to some embodiments.

FIG. 28B shows the scene of FIG. 27 represented by inner rectangularplanes, according to some embodiments.

FIG. 28C shows the scene of FIG. 27 represented by polygon planes,according to some embodiments.

FIG. 29 shows the scene of FIG. 27 with denoised mesh by planarizing themesh shown in FIG. 27, according to some embodiments.

FIG. 30 is a flow chart illustrating a method of generating a model ofan environment represented by a mesh, according to some embodiments.

FIG. 31 is a schematic diagram illustrating a 2D representation of aportion of a physical world by four blocks, according to someembodiments.

FIGS. 32A-32D are schematic diagrams illustrating a mesh evolution of anexemplary mesh block during a multi-stage simplification, according tosome embodiments.

FIGS. 33A and 33B show representations of the same environment withoutsimplification and with simplification through triangle reduction,respectively.

FIGS. 34A and 34B show close-up representations of the same environmentwithout simplification and with simplification through trianglereduction, respectively.

FIGS. 35A and 35B show representations of the same environment withoutplanarization and with planarization, respectively.

FIGS. 36A and 36B show representations of the same environment withoutsimplification and with simplification through removal of disconnectedcomponents, respectively.

FIG. 37 is a schematic diagram illustrating an electronic system thatenables interactive X reality environments for multiple users, accordingto some embodiments.

FIG. 38 is a schematic diagram, illustrating interaction of componentsof the electronic system in FIG. 37, according to some embodiments.

FIG. 39 is a flow chart, illustrating a method of operating theelectronic system in FIG. 37, according to some embodiments.

FIG. 40 is a flow chart, illustrating a method of capturing 3Dinformation about objects in a physical world and representing thephysical world as blocks of 3D reconstruction in FIG. 39, according tosome embodiments.

FIG. 41 is a flow chart, illustrating a method of selecting versions ofblocks representing the subset of blocks in FIG. 39, according to someembodiments.

FIG. 42 is a flow chart, illustrating a method of operating theelectronic system in FIG. 37, according to some embodiments.

FIG. 43A is a simplified schematic diagram illustrating an update beingdetected in a portion of a physical world represented by mesh blocks,according to some embodiments.

FIG. 43B is a simplified schematic diagram illustrating a mesh block,according to some embodiments.

FIG. 43C is a simplified schematic diagram illustrating a crack at edgesof two adjacent mesh blocks, according to some embodiments.

FIG. 43D is a simplified schematic diagram illustrating the crack inFIG. 43C being papered over by implementing mesh skirts that overlap anadjacent mesh blocks, according to some embodiments.

FIG. 44 is a schematic diagram, illustrating a 2D representation of aportion of a physical world by four blocks, according to someembodiments.

FIG. 45 is a schematic diagram, illustrating a 3D representation of aportion of a physical world by eight blocks, according to someembodiments.

FIG. 46 is a schematic diagram, illustrating a 3D representation of aportion of a physical world obtained by updating the 3D representationin FIG. 45, according to some embodiments.

FIG. 47 is a schematic diagram, illustrating an example of an augmentedworld viewable by first and second users wearing AR display systems,according to some embodiments.

FIG. 48 is a schematic diagram, illustrating an example of an augmentedworld obtained by updating the augmented world of FIG. 47 with newversions of blocks, according to some embodiments.

FIG. 49 is a schematic diagram illustrating an occlusion renderingsystem, according to some embodiments.

FIG. 50 is a schematic diagram illustrating a depth image with holes.

FIG. 51 is a flow chart illustrating a method of occlusion rendering inan augmented reality environment, according to some embodiments.

FIG. 52 is a flow chart illustrating details of generating surfaceinformation from depth information captured by a depth sensor worn by auser in FIG. 51, according to some embodiments.

FIG. 53 is a flow chart illustrating details of filtering the depthinformation to generate a depth map in FIG. 52, according to someembodiments.

FIG. 54A is a sketch of a region being imaged with a depth camera from afirst point of view to identify voxels that are occupied by a surfaceand those that are empty.

FIG. 54B is a sketch of a region being imaged with a depth camera frommultiple points of view to identify voxels that are occupied by asurface and are empty, and indicating a “hole”, for which no volumetricinformation is available, as a result of voxels in the region of the“hole” not having been imaged with the depth camera.

DETAILED DESCRIPTION

Described herein are methods and apparatus for creating and using athree-dimensional (3D) world reconstruction in an augmented reality(AR), mixed reality (MR), or virtual reality (VR) system. To providerealistic AR/MR/VR experiences to users, the AR/MR/VR system must knowthe user's physical surroundings in order to correctly correlate alocation of virtual objects in relation to real objects. The worldreconstruction may be constructed from image and depth information aboutthose physical surroundings that are collected with sensors that arepart of the AR/MR/VR system. The world reconstruction may then be usedby any of multiple components of such a system. For example, the worldreconstruction may be used by components that perform visual occlusionprocessing, compute physics-based interactions or perform environmentalreasoning.

Occlusion processing identifies portions of a virtual object that shouldnot be rendered for and/or displayed to a user because there is anobject in the physical world blocking that user's view of the locationwhere that virtual object is to be perceived by the user. Physics-basedinteractions are computed to determine where or how a virtual objectappears to the user. For example, a virtual object may be rendered so asto appear to be resting on a physical object, moving through empty spaceor colliding with a surface of a physical object. The worldreconstruction provides a model from which information about objects inthe physical world may be obtained for such calculations.

Environmental reasoning may also use the world reconstruction in thecourse of generating information that can be used in computing how torender virtual objects. For example, environmental reasoning may involveidentifying clear surfaces by recognizing that they are window panes orglass table tops. From such an identification, regions that containphysical objects might be classified as not occluding virtual objectsbut might be classified as interacting with virtual objects.Environmental reasoning may also generate information used in otherways, such as identifying stationary objects that may be trackedrelative to a user's field of view to compute motion of the user's fieldof view.

However, there are significant challenges in providing such a system.Substantial processing may be required to compute the worldreconstruction. Further, the AR/MR/VR systems must correctly know how toposition virtual objects in relation to the user's head, body, etc. Asthe user's position in relation to the physical environment changes, therelevant portions of the physical world can also change, which canrequire further processing. Moreover, the 3D reconstruction data areoften required to be updated as objects move in the physical world(e.g., a cup moves on a table). Updates to the data representing theenvironment that the user is experiencing must be performed quicklywithout using so much of the computing resources of the computergenerating the AR/MR/VR environment because it is unable to performother functions while performing world reconstruction. Further, theprocessing of reconstruction data by components that “consume” that datacan exacerbate the demands on computer resources.

Known AR/MR/VR systems require high computing power (e.g., a GPU) to runreal-time World Reconstruction only within a pre-defined reconstructionvolume (e.g., a predefined voxel grid). The inventors have realized andappreciated techniques for operating AR/MR/VR systems to provideaccurate 3D reconstruction data in real-time with low usage ofcomputational resources, such as compute power (e.g., a single ARMcore), memory (e.g., less than 1 GB), and network bandwidth (e.g., lessthan 100 Mbps). These techniques relate to reducing processing requiredto generate and maintain the world reconstruction as well as toproviding and consuming data with low computational overhead.

These techniques may include reducing the amount of data that isprocessed when updating a world reconstruction, such as by identifyingportions of sensor data available at any time to use in the creating orupdating the world reconstruction. Sensor data may be selected, forexample, based on whether it represents a portion of the physical worldthat is likely near a surface of an object to the represented in theworld reconstruction.

In some embodiments, computational resources may be reduced bysimplifying the data representing the world reconstruction. A simplerrepresentation may reduce resources for the processing, storage and/ormanagement of that data as well as for its use.

In some embodiments, use of computational resources may be reduced byrepresenting the physical world in blocks that may be stored andretrieved separately, but combined in a way that provides a realisticrepresentation of the physical world. The blocks may be managed inmemory to limit computational resources and may, in some embodiments,enable sharing of blocks across AR/MR/VR systems operating in the samephysical space such that each AR/MR/VR system does less processing toconstruct a world reconstruction.

In some embodiments, the computational burden may be lessened and/oraccuracy of processing information representing the physical world maybe increased by associating altitude information with representations ofportions of the physical world. Those representations may be, for adense representation of the physical world, mesh blocks as describedherein, or a connected mesh representing a larger portion of thephysical world, such as a floor of a building.

In some embodiments, use of computational resources may be reduced byselecting from among different representations of the physical worldwhen accessing information about the physical world. The worldreconstruction, for example, may include information about the physicalworld captured from different sensors and/or stored in differentformats. The data that is the simplest to consume or provide may besupplied to a component using the world reconstruction to render virtualobjects. Where simpler data is unavailable, data acquired with adifferent sensor, which may generate a higher computation load, may beaccessed. As an example, the world reconstruction may include a depthmap collected with a depth sensor and a more fulsome representation ofthe 3D world, such as may be stored as a mesh computed from imageinformation. Information about the physical world may be supplied to acomponent doing occlusion processing based on the depth map where it isavailable. Where there are holes in the depth map, information to fillthose holes may be extracted from the mesh. In some embodiments, thedepth map may be “live,” representing the physical world as captured bythe depth sensor at the time the data is accessed.

Techniques as described herein may be used together or separately withmany types of devices and for many types of scenes, including wearableor portable devices with limited computations resources that provide anaugmented reality scene.

AR System Overview

FIGS. 1-2 illustrate such scenes. For purposes of illustration, an ARsystem is used as an example of an XR system. FIGS. 3-8 illustrate anexemplary AR system, including one or more processors, memory, sensorsand user interfaces that may operate according to the techniquesdescribed herein.

Referring to FIG. 1, an AR scene 4 is depicted wherein a user of an ARtechnology sees a physical world park-like setting 6, featuring people,trees, buildings in the background, and a concrete platform 8. Inaddition to these items, the user of the AR technology also perceivesthat they “see” a robot statue 10 standing upon the physical worldconcrete platform 8, and a cartoon-like avatar character 2 flying bywhich seems to be a personification of a bumble bee, even though theseelements (e.g., the avatar character 2, and the robot statue 10) do notexist in the physical world. Due to the extreme complexity of the humanvisual perception and nervous system, it is challenging to produce an ARtechnology that facilitates a comfortable, natural-feeling, richpresentation of virtual image elements amongst other virtual or physicalworld imagery elements.

Such an AR scene may be achieved with a system that include a worldreconstruction component, which may build and update a representation ofthe physical world surfaces around the user. This representation may beused to occlude rendering, to place virtual objects, in physics basedinteractions, and for virtual character path planning and navigation, orfor other operations in which information about the physical world isused. FIG. 2 depicts another example of an AR scene 200, showingexemplary world reconstruction use cases, including visual occlusion202, physics-based interactions 204, and environment reasoning 206,according to some embodiments.

The exemplary scene 200 is a living room having walls, a book shelf onone side of a wall, a floor lamp at a corner of the room, a floor, asofa and coffee table on the floor. In addition to these physical items,the user of the AR technology also perceives virtual objects such asimages on the wall behind the sofa, birds flying through the door, adeer peeking out from the book shelf, and a decoration in the form of awindmill placed on the coffee table. For the images on the wall, the ARtechnology requires information about not only surfaces of the wall butalso objects and surfaces in the room such as lamp shape, which areoccluding the images to render the virtual objects correctly. For theflying birds flying, the AR technology requires information about allthe objects and surfaces around the room for rendering the birds withrealistic physics to avoid the objects and surfaces or bounce off themif the birds collide. For the deer, the AR technology requiresinformation about the surfaces such as the floor or coffee table tocompute where to place the deer. For the windmill, the system mayidentify that is an object separate from the table and may reason thatit is movable, whereas corners of shelves or corners of the wall may bereasoned to be stationary. Such a distinction may be used in reasoningas to which portions of the scene are used or updated in each of variousoperations.

A scene may be presented to the user via a system that includes multiplecomponents, including a user interface that can stimulate one or moreuser senses, including sight sound and/or touch. In addition, the systemmay include one or more sensors that may measure parameters of thephysical portions of the scene, including position and/or motion of theuser within the physical portions of the scene. Further, the system mayinclude one or more computing devices, with associated computerhardware, such as memory. These components may be integrated into asingle device or more be distributed across multiple interconnecteddevices. In some embodiments some or all of these components may beintegrated into a wearable device.

FIG. 3 depicts an AR system 302 configured to provide an experience ofAR contents interacting with a physical world 306, according to someembodiments. The AR system 302 may include a display 308. In theillustrated embodiment, the display 308 may be worn by the user as partof a headset such that a user may wear the display over their eyes likea pair of goggles or glasses. At least a portion of the display may betransparent such that a user may observe a see-through reality 310. Thesee-through reality 310 may correspond to portions of the physical world306 that are within a present viewpoint of the AR system 302, which maycorrespond to the viewpoint of the user in the case that the user iswearing a headset incorporating both the display and sensors of the ARsystem to acquire information about the physical world.

AR contents may also be presented on the display 308, overlaid on thesee-through reality 310. To provide accurate interactions between ARcontents and the see-through reality 310 on the display 308, the ARsystem 302 may include sensors 322 configured to capture informationabout the physical world 306.

The sensors 322 may include one or more depth sensors that output depthmaps 312. Each depth map 312 may have multiple pixels, each of which mayrepresent a distance to a surface in the physical world 306 in aparticular direction relative to the depth sensor. Raw depth data maycome from a depth sensor to create a depth map. Such depth maps may beupdated as fast as the depth sensor can form a new image, which may behundreds or thousands of times per second. However, that data may benoisy and incomplete, and have holes shown as black pixels on theillustrated depth map.

The system may include other sensors, such as image sensors. The imagesensors may acquire information that may be processed to represent thephysical world in other ways. In some embodiments, the system mayinclude other sensors such as, for example, one or more of thefollowing: infrared camera sensors, visible spectrum camera sensors,structured light emitters and/or sensors, infrared light emitters,coherent light emitters and/or sensors, gyroscopic sensors,accelerometers, magnetometers, altimeters, proximity sensors, GPSsensors, ultrasonic emitters and detectors, and haptic interfaces.

The sensor data may be used on a portable system to enable the system totrack its location with respect to the physical world. Further, thatinformation may enable the device to localize with respect to acoordinate frame that is persisted across sessions of the system and/orshared with other systems. The ability to determine location withrespect to a shared coordinate frame is described for example, inapplication U.S. App. No. 62/982,694, which is hereby incorporated byreference in its entirety. In that application, location is determinedbased on sparse representations of the physical world, such as acanonical map against which each of multiple portable user systems maylocalize by providing a localization request including sparseinformation about a portion of the current environment of the portablesystem.

Dense data about the physical world may also be acquired. The sensordata, for example, may be processed in world reconstruction component316 to create a mesh, representing connected portions of objects in thephysical world. Metadata about such objects, including for example,color and surface texture, may similarly be acquired with the sensorsand stored as part of the world reconstruction. Metadata about alocation of a device including the system may be determined or inferredbased on the sensor data. For example, localization with respect to acanonical map may be used to determine location. Alternatively oradditionally, magnetometers, altimeters, GPS sensors, and the like maybe used to determine or infer the location of the device. The locationof the device may be of varying degrees of granularity. For example, theaccuracy of the determined location of the device may vary from coarse(e.g., accurate to a 10 meter diameter sphere), to fine (e.g., accurateto a 3 meter diameter sphere), to extra fine (e.g., accurate to a 1meter diameter sphere), to ultra fine (e.g., accurate to a 0.5 meterdiameter sphere), etc.

In some embodiments, the sensor data may be processed to determine whichfloor of a multi-story building the device is located in. As a user ofthe device navigates about one or more floors of the multi-storybuilding, the sensor data may be processed in world reconstructioncomponent 316 to create a mesh representing connected portions ofobjects in the physical world, which may be on different floors. Forexample, as the user of the device navigates about one or more floors ofthe multi-story building, the sensor data may be processed in worldreconstruction component 316 to create a continuous mesh that includesobjects from the various floors that the user navigated about, orobjects that are otherwise at different altitudes. Accordingly, in someembodiments, altitude and floor information may be used interchangeably,with a relative difference in altitude indicating a difference infloors, or vice versa. Alternatively or additionally, the mesh may havemultiple layers, with each layer representing one or more floors in thebuilding or otherwise representing objects at a different altitude.

In some embodiments, altitude may be measured directly, such as with analtimeter. Relative changes in altitude may enable a device to detectthat it is moving between floors such that mesh data used or generatedby the device is associated with an appropriate floor at any given time.

Alternatively or additionally, altitude may be inferred from othermeasurements. For example, in some embodiments of an XR system, anenvironment, such as an office building, may have markers identifying afloor of a building, which may be recognized by processing on the sensoroutput on a portable system. The markers, for example, may be in theform of a machine readable QR code or may be color of walls or otherindicator that may be detected by a sensor. As another example, aportable system may be configured to read signs or other human-readableindicators of location, such as elevator buttons. As yet a furtherexample, canonical maps may be created for each floor of a building,such that localization with respect to a map uniquely representing aparticular floor may indicate altitude. In embodiments in which multiplefloors of a building have similar layout, other indictors may be usedinstead of or in addition to a localization results. As yet a furtherexample, as the device tracks its position with respect to the physicalworld, it may detect actions that indicate moving between floors in abuilding. Such detected actions may include traversing a set of stairsor an escalator.

Incorporating altitude information into the dense and/or sparserepresentation of an environment may enable more robust processing ofthe sparse and/or dense information. For example, in hotel or officebuilding with many floors that look similar, conventional localizationapproaches may lead to inaccurate results and/or require many iterationsto resolve location between similar-looking floors. Incorporatingaltitude information into the sparse and/or dense representations of thephysical world may increase accuracy and/or reduce processing timerequired to determine a localization result, as the localization servicemay limit localization processing by only attempting localization withrespect to maps representing portions of the physical world with thesame altitude as the current altitude of the portable system. Inembodiments in which localization is performed by a localizationservice, a localization request may include altitude information.

As another example, associating altitude information with meshinformation may enable vertical ordering of the dense information suchthat processing that more accurately determine surfaces between a userand an object in a direction that the user may be looking. For example,processing for a user on an upper floor of a building may determine thata virtual object on a floor below the user is not visible because it isoccluded by the floor. In another scenario, processing may determinethat the virtual object is visible because the user is looking down at astairwell.

The system may also acquire information about the headpose of the userwith respect to the physical world. In some embodiments, sensors 310 mayinclude inertial measurement units that may be used to compute and/ordetermine a headpose 314. A headpose 314 for a depth map may indicate apresent viewpoint of a sensor capturing the depth map with six degreesof freedom (6DoF), for example, but the headpose 314 may be used forother purposes, such as to relate image information to a particularportion of the physical world or to relate the position of the displayworn on the user's head to the physical world. In some embodiments, theheadpose information may be derived in other ways than from an IMU, suchas from analyzing objects in an image.

The world reconstruction component 316 may receive the depth maps 312and headposes 314, and any other data from the sensors, and integratethat data into a reconstruction 318, which may at least appears to be asingle, combined reconstruction. The reconstruction 318 may be morecomplete and less noisy than the sensor data. The world reconstructioncomponent 316 may update the reconstruction 318 using spatial andtemporal averaging of the sensor data from multiple viewpoints overtime.

The reconstruction 318 may include representations of the physical worldin one or more data formats including, for example, voxels, meshes,planes, etc. The different formats may represent alternativerepresentations of the same portions of the physical world or mayrepresent different portions of the physical world. In the illustratedexample, on the left side of the reconstruction 318, portions of thephysical world are presented as a global surface; on the right side ofthe reconstruction 318, portions of the physical world are presented asmeshes.

The reconstruction 318 may be used for AR functions, such as producing asurface representation of the physical world for occlusion processing orphysics-based processing. This surface representation may change as theuser moves or objects in the physical world change. Aspects of thereconstruction 318 may be used, for example, by a component 320 thatproduces a changing global surface representation in world coordinates,which may be used by other components.

The AR contents may be generated based on this information, such as byAR applications 304. An AR application 304 may be a game program, forexample, that performs one or more functions based on information aboutthe physical world, such visual occlusion, physics-based interactions,and environment reasoning. It may perform these functions by queryingdata in different formats from the reconstruction 318 produced by theworld reconstruction component 316. In some embodiments, component 320may be configured to output updates when a representation in a region ofinterest of the physical world changes. That region of interest, forexample, may be set to approximate a portion of the physical world inthe vicinity of the user of the system, such as the portion within theview field of the user, or is projected (predicted/determined) to comewithin the view field of the user.

The AR applications 304 may use this information to generate and updatethe AR contents. The virtual portion of the AR contents may be presentedon the display 308 in combination with the see-through reality 310,creating a realistic user experience.

In some embodiments, an AR experience may be provided to a user througha wearable display system. FIG. 4 illustrates an example of wearabledisplay system 80 (hereinafter referred to as “system 80”). The system80 includes a head mounted display device 62 (hereinafter referred to as“display device 62”), and various mechanical and electronic modules andsystems to support the functioning of the display device 62. The displaydevice 62 may be coupled to a frame 64, which is wearable by a displaysystem user or viewer 60 (hereinafter referred to as “user 60”) andconfigured to position the display device 62 in front of the eyes of theuser 60. According to various embodiments, the display device 62 may bea sequential display. The display device 62 may be monocular orbinocular. In some embodiments, the display device 62 may be an exampleof the display 308 in FIG. 3.

In some embodiments, a speaker 66 is coupled to the frame 64 andpositioned proximate an ear canal of the user 60. In some embodiments,another speaker, not shown, is positioned adjacent another ear canal ofthe user 60 to provide for stereo/shapeable sound control. The displaydevice 62 is operatively coupled, such as by a wired lead or wirelessconnectivity 68, to a local data processing module 70 which may bemounted in a variety of configurations, such as fixedly attached to theframe 64, fixedly attached to a helmet or hat worn by the user 60,embedded in headphones, or otherwise removably attached to the user 60(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local data processing module 70 may include a processor, as well asdigital memory, such as non-volatile memory (e.g., flash memory), bothof which may be utilized to assist in the processing, caching, andstorage of data. The data include data a) captured from sensors (whichmay be, e.g., operatively coupled to the frame 64) or otherwise attachedto the user 60, such as image capture devices (such as cameras),microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, and/or gyros; and/or b) acquired and/or processedusing remote processing module 72 and/or remote data repository 74,possibly for passage to the display device 62 after such processing orretrieval. The local data processing module 70 may be operativelycoupled by communication links 76, 78, such as via a wired or wirelesscommunication links, to the remote processing module 72 and remote datarepository 74, respectively, such that these remote modules 72, 74 areoperatively coupled to each other and available as resources to thelocal processing and data module 70. In some embodiments, the worldreconstruction component 316 in FIG. 3 may be at least partiallyimplemented in the local data processing module 70. For example, thelocal data processing module 70 may be configured to execute computerexecutable instructions to generate the physical world representationsbased at least in part on at least a portion of the data.

In some embodiments, the local data processing module 70 may include oneor more processors (e.g., a graphics processing unit (GPU)) configuredto analyze and process data and/or image information. In someembodiments, the local data processing module 70 may include a singleprocessor (e.g., a single-core or multi-core ARM processor), which wouldlimit the module 70's compute budget but enable a more miniature device.In some embodiments, the world reconstruction component 316 may use acompute budget less than a single ARM core to generate physical worldrepresentations in real-time on a non-predefined space such that theremaining compute budget of the single ARM core can be accessed forother uses such as, for example, extracting meshes.

In some embodiments, the remote data repository 74 may include a digitaldata storage facility, which may be available through the Internet orother networking configuration in a “cloud” resource configuration. Insome embodiments, all data is stored and all computations are performedin the local data processing module 70, allowing fully autonomous usefrom a remote module. A world reconstruction, for example, may be storedin whole or in part in this repository 74.

In some embodiments, the local data processing module 70 is operativelycoupled to a battery 82. In some embodiments, the battery 82 is aremovable power source, such as over the counter batteries. In otherembodiments, the battery 82 is a lithium-ion battery. In someembodiments, the battery 82 includes both an internal lithium-ionbattery chargeable by the user 60 during non-operation times of thesystem 80 and removable batteries such that the user 60 may operate thesystem 80 for longer periods of time without having to be tethered to apower source to charge the lithium-ion battery or having to shut thesystem 80 off to replace batteries.

FIG. 5A illustrates a user 30 wearing an AR display system rendering ARcontent as the user 30 moves through a physical world environment 32(hereinafter referred to as “environment 32”). The user 30 positions theAR display system at positions 34, and the AR display system recordsambient information of a passable world (e.g., a digital representationof the real objects in the physical world that can be stored and updatedwith changes to the real objects in the physical world) relative to thepositions 34 such as pose relation to mapped features or directionalaudio inputs. The positions 34 are aggregated to data inputs 36 andprocessed at least by a passable world module 38, which may beimplemented, for example, by processing on a remote processing module 72of FIG. 3. In some embodiments, the passable world module 38 may includethe world reconstruction component 316.

The passable world module 38 determines where and how AR content 40 canbe placed in the physical world as determined from the data inputs 36.The AR content is “placed” in the physical world by presenting via theuser interface both a representation of the physical world and the ARcontent, with the AR content rendered as if it were interacting withobjects in the physical world and the objects in the physical worldpresented as if the AR content were, when appropriate, obscuring theuser's view of those objects. In some embodiments, the AR content may beplaced by appropriately selecting portions of a fixed element 42 (e.g.,a table) from a reconstruction (e.g., the reconstruction 318) todetermine the shape and position of the AR content 40. As an example,the fixed element may be a table and the virtual content may bepositioned such that it appears to be on that table. In someembodiments, the AR content may be placed within structures in a fieldof view 44, which may be a present field of view or an estimated futurefield of view. In some embodiments, the AR content may be placedrelative to a mapped mesh model 46 of the physical world.

As depicted, the fixed element 42 serves as a proxy for any fixedelement within the physical world which may be stored in the passableworld module 38 so that the user 30 can perceive content on the fixedelement 42 without the system having to map to the fixed element 42 eachtime the user 30 sees it. The fixed element 42 may, therefore, be amapped mesh model from a previous modeling session or determined from aseparate user but nonetheless stored on the passable world module 38 forfuture reference by a plurality of users. Therefore, the passable worldmodule 38 may recognize the environment 32 from a previously mappedenvironment and display AR content without a device of the user 30mapping the environment 32 first, saving computation process and cyclesand avoiding latency of any rendered AR content.

The mapped mesh model 46 of the physical world may be created by the ARdisplay system and appropriate surfaces and metrics for interacting anddisplaying the AR content 40 can be mapped and stored in the passableworld module 38 for future retrieval by the user 30 or other userswithout the need to re-map or model. In some embodiments, the datainputs 36 are inputs such as geolocation, user identification, andcurrent activity to indicate to the passable world module 38 which fixedelement 42 of one or more fixed elements are available, which AR content40 has last been placed on the fixed element 42, and whether to displaythat same content (such AR content being “persistent” content regardlessof user viewing a particular passable world model).

FIG. 5B illustrates a schematic of a viewing optics assembly 48 andattendant components. Oriented to user eyes 49, in some embodiments, twoeye tracking cameras 50 detect metrics of the user eyes 49 such as eyeshape, eyelid occlusion, pupil direction and glint on the user eyes 49.In some embodiments, a depth sensor 51, such as a time of flight sensor,emits relay signals to the world to determine distance to given objects.In some embodiments, world cameras 52 record a greater-than-peripheralview to map the environment 32 and detect inputs that may affect ARcontent. Camera 53 may further capture a specific timestamp of physicalworld images within a field of view of the user. Each of the worldcameras 52, the camera 53 and the depth sensor 51 have respective fieldsof view of 54, 55, and 56 to collect data from and record a physicalworld scene, such as physical world environment 32 depicted in FIG. 3A.

Inertial measurement units 57 may determine movement and orientation ofthe viewing optics assembly 48. In some embodiments, each component isoperatively coupled to at least one other component. For example, thedepth sensor 51 is operatively coupled to the eye tracking cameras 50 asa confirmation of measured accommodation against actual distance theuser eyes 49 are looking at.

Information from these sensors in viewing optics assembly 48 may becoupled to one or more of the processors in the system. The processorsmay generate data that may be rendered so as to cause the user toperceive virtual content interacting with objects in the physical world.That rendering may be implemented in any suitable way, includinggenerating image data that depicts both physical and virtual objects. Inother embodiments, physical and virtual content may be depicted in onescene by modulating the opacity of a display device that a user looksthrough at the physical world. The opacity may be controlled so as tocreate the appearance of the virtual object and also to block the userfrom seeing objects in the physical world that are occluded by thevirtual objects. Regardless of how content is presented to a user, amodel of the physical world is required so that characteristics of thevirtual objects, which can be impacted by physical objects, includingthe shape, position, motion and visibility of the virtual object, can becorrectly computed. In some embodiments, the model may include thereconstruction of a physical world, for example, the reconstruction 318.

That model may be created from data collected from sensors on a wearabledevice of the user. Though, in some embodiments, the model may becreated from data collected by multiple users, which may be aggregatedin a computing device remote from all of the users (and which may be “inthe cloud”).

The model may be created, at least in part, by a world reconstructionsystem, for example, the world reconstruction component 316 of FIG. 3depicted in more detail in FIG. 6. The world reconstruction component316 may include a perception module 160 that may generate, update, andstore representations for a portion of the physical world. In someembodiments, the perception module 160 may represent the portion of thephysical world within a reconstruction range of the sensors as multiplevoxels. Each voxel may correspond to a 3D cube of a predetermined volumein the physical world, and include surface information, indicatingwhether there is a surface in the volume represented by the voxel.Voxels may be assigned values indicating whether their correspondingvolumes have been determined to include surfaces of physical objects,determined to be empty or have not yet been measured with a sensor andso their value is unknown. It should be appreciated that valuesindicating that voxels that are determined to be empty or unknown neednot be explicitly stored, as the values of voxels may be stored incomputer memory in any suitable way, including storing no informationfor voxels that are determined to be empty or unknown.

FIG. 7A depicts an example of a 3D space 100 discretized into voxels102. In some embodiments, the perception module 160 may determineobjects of interest and set the volume of a voxel in order to capturefeatures of the objects of interest and avoid redundant information. Forexample, the perception module 160 may be configured to identify largerobjects and surfaces, such as walls, ceilings, floors, and largefurniture. Accordingly, a volume of a voxel may be set to a relativelylarge size, for example, a cube of 4 cm³.

A reconstruction of a physical world including voxels may be referred toas a volumetric model. Information to create a volumetric model may becreated over time as the sensors move about the physical world. Suchmotion may happen as the user of a wearable device including the sensorsmoves around. FIGS. 8A-F depict an example of reconstructing a physicalworld into a volumetric model. In the illustrated example, the physicalworld includes a portion 180 of a surface which is shown in FIG. 8A. InFIG. 8A, a sensor 182 at a first location may have a field of view 184,within which the portion 180 of the surface is visible.

The sensor 182 may be of any suitable type, such as a depth sensor.However, depth data may be derived from an image sensor(s) or in otherways. The perception module 160 may receive data from the sensor 182,and then set the values of multiple voxels 186 as illustrated in FIG. 8Bto represent the portion 180 of the surface visible by the sensor 182 inthe field of view 184.

In FIG. 8C, the sensor 182 may move to a second location and have afield of view 188. As shown in FIG. 8D, a further group of voxels becomevisible, and the values of these voxels may be set to indicate thelocation of the portion of the surface that has entered the field ofview 188 of sensor 182. The values of these voxels may be added to thevolumetric model for the surface

In FIG. 8E, the sensor 182 may further move to a third location and havea field of view 190. In the illustrated example, additional portions ofthe surface becomes visible in the field of view 190. As shown in FIG.8F, a further group of voxels may become visible, and the values ofthese voxels may be set to indicate the location of the portion of thesurface that has entered the field of view 190 of the sensor 182. Thevalues of these voxels may be added to the volumetric model for thesurface. As shown in FIG. 6, this information may be stored as part ofthe persisted world as volumetric information 162 a. Information aboutthe surfaces may also be stored, such as color or texture. Suchinformation may be stored, for example, as volumetric metadata 162 b.

In addition to generating information for a persisted worldrepresentation, the perception module 160 may identify and outputindications of changes in a region around a user of a AR system.Indications of such changes may trigger updates to volumetric datastored as part of the persisted world, or trigger other functions, suchas triggering components 304 that generate AR content to update the ARcontent.

In some embodiments, the perception module 160 may identify changesbased on a signed distance function (SDF) model. The perception module160 may be configured to receive sensor data such as, for example, depthmaps 160 a and headposes 160 b, and then fuse the sensor data into a SDFmodel 160 c. Depth maps 160 a may provide SDF information directly, andimages may be processed to arrive at SDF information. The SDFinformation represents distance from the sensors used to capture thatinformation. As those sensors may be part of a wearable unit, the SDFinformation may represent the physical world from the perspective ofwearable unit and therefore the perspective of the user. The headposes160 b may enable the SDF information to be related to a voxel in thephysical world.

Referring back to FIG. 6, in some embodiments, the perception module 160may generate, update, and store representations for the portion of thephysical world that is within a perception range. The perception rangemay be determined based, at least in part, on a sensor's reconstructionrange, which may be determined based, at least in part, on the limits ofa sensor's observation range. As a specific example, an active depthsensor that operates using active IR pulses may be operate reliably overa range of distances, creating the observation range of the sensor,which may be from a few centimeters or tens of centimeters to a fewmeters.

FIG. 7B depicts a reconstruction range with respect to a sensor 104having a viewpoint 106. A reconstruction of 3D spaces within theviewpoint 106 may be built based on data captured by the sensor 104. Inthe illustrated example, the sensor 104 has an observation range of 40cm to 5 m. In some embodiments, a sensor's reconstruction range may bedetermined to be smaller than the observation range of the sensorbecause sensor outputs close to its observation limits may be morenoisy, incomplete, and inaccurate. For example, in the illustratedexample of 40 cm to 5 m, a corresponding reconstruction range may be setto be from 1 to 3 m, and data collected with the sensor indicatingsurfaces outside this range may not be used.

In some embodiments, the perception range may be larger than a sensor'sreconstruction range. If components 164 that use data about the physicalworld require data about regions within the perception range that areoutside the portions of the physical world that are within the currentreconstruction range, that information may be provided from thepersisted world 162. Accordingly, information about the physical worldmay be readily accessible by a query. In some embodiments, an API may beprovided to respond to such a query, providing information about thecurrent perception range of the user. Such technique may reduce timeneeded to access an existing reconstruction and provide an improved userexperience.

In some embodiments, the perception range may be a 3D spacecorresponding to a bounding box centered around a user location. As theuser moves, the portion of the physical world within the perceptionrange, which may be queriable by the components 164, may move with theuser. FIG. 7C depicts a bounding box 110 centered around a location 112.It should be appreciated that the size of the bounding box 110 may beset to enclose a sensor's observation range with reasonable extensionsbecause a user cannot move at an unreasonable speed. In the illustratedexample, a sensor worn by the user has an observation limit of 5 m. Thebounding box 110 is set as a cube of 20 m³.

Referring back to FIG. 6, the world reconstruction component 316 mayinclude additional modules that may interact with the perception module160. In some embodiments, a persisted world module 162 may receiverepresentations for the physical world based on data acquired by theperception module 160. The persisted world module 162 also may includevarious formats of representations of the physical world. For example,volumetric metadata 162 b such as voxels may be stored as well as meshes162 c and planes 162 d. In some embodiments, other information, such asdepth maps could be saved.

In some embodiments, the perception module 160 may include modules thatgenerate representations for the physical world in various formatsincluding, for example, meshes 160 d, planes and semantics 160 e. Thesemodules may generate representations based on data within the perceptionrange of one or more sensors at the time the representation is generatedas well as data captured at prior times and information in the persistedworld 162. In some embodiments, these components may operate on depthinformation captured with a depth sensor. However, the AR system mayinclude vision sensors and may generate such representations byanalyzing monocular or binocular vision information.

In some embodiments, these modules may operate on regions of thephysical world, such as regions represented by blocks or tiles, asdescribed below. Those modules may be triggered to update a block ortile, or other subregion of the physical world, when the perceptionmodule 160 detects a change in the physical world in that subregion.Such a change, for example, may be detected by detecting a new surfacein the SDF model 160 c or other criteria, such as changing the value ofa sufficient number of voxels representing the subregion.

The world reconstruction component 316 may include components 164 thatmay receive representations of the physical world from the perceptionmodule 160. Information about the physical world may be pulled by thesecomponents according to, for example, a use request from an application.In some embodiments, information may be pushed to the use components,such as via an indication of a change in a pre-identified region or achange of the physical world representation within the perception range.The components 164, may include, for example, game programs and othercomponents that perform processing for visual occlusion, physics-basedinteractions, and environment reasoning.

Responding to the queries from the components 164, the perception module160 may send representations for the physical world in one or moreformats. For example, when the component 164 indicates that the use isfor visual occlusion or physics-based interactions, the perceptionmodule 160 may send a representation of surfaces. When the component 164indicates that the use is for environmental reasoning, the perceptionmodule 160 may send meshes, planes and semantics of the physical world.

In some embodiments, the perception module 160 may include componentsthat format information to provide the component 164. An example of sucha component may be raycasting component 160 f. A use component (e.g.,component 164), for example, may query for information about thephysical world from a particular point of view. Raycasting component 160f may select from one or more representations of the physical world datawithin a field of view from that point of view.

Viewpoint Dependent Brick Selection for Fast Volumetric Reconstruction

As should be appreciated from the foregoing description, the perceptionmodule 160, or another component of an AR system, may process data tocreate 3D representations of portions of the physical world. Data to beprocessed may be reduced by culling parts of a 3D reconstruction volumebased at last in part on a camera frustum and/or depth image, extractingand persisting plane data, capturing, persisting and updating 3Dreconstruction data in blocks that allow local update while maintainingneighbor consistency, providing occlusion data to applicationsgenerating such scenes, where the occlusion data is derived from acombination of one or more depth data sources, and/or performing amulti-stage mesh simplification.

A world reconstruction system may integrate sensor data over time frommultiple viewpoints of a physical world. The poses of the sensors (e.g.,position and orientation) may be tracked as a device including thesensors is moved. As the sensor's frame pose is known and how it relatesto the other poses, each of these multiple viewpoints of the physicalworld may be fused together into a single, combined reconstruction. Thereconstruction may be more complete and less noisy than the originalsensor data by using spatial and temporal averaging (i.e. averaging datafrom multiple viewpoints over time).

The reconstruction may contain data of different levels ofsophistication including, for example, raw data such as live depth data,fused volumetric data such as voxels, and computed data such as meshes.

In some embodiments, AR and MR systems represent a 3D scene with aregular voxel grid, where each voxel may contain a signed distance field(SDF) value. The SDF value describes whether the voxel lies inside oroutside a surface in the scene to be reconstructed and a distance fromthe voxel to the surface. Computing the 3D reconstruction data,representing the desired volume of the scene, requires a large amount ofmemory and processing power. These requirements increase for scenes thatrepresents larger spaces as the number of variables required for the 3Dreconstruction grows cubically with the number of depth imagesprocessed.

Described herein is an efficient way of reducing processing. Inaccordance with some embodiments, a scene may be represented by one ormore bricks. Each brick may include multiple voxels. The bricksprocessed to generate the 3D reconstruction of the scene may be selectedby culling the set of bricks representing the scene based on a frustumderived from the field-of-view (FOV) of an image sensor, and/or a depthimage (or “depth map”) of the scene created with a depth sensor.

The depth image may have one or more pixels, each representing adistance to a surface in the scene. These distances can be related to aposition relative to an image sensor, such that the data output from theimage sensor may be selectively processed. Image data may be processedfor those bricks representing portions of the 3D scene that containsurfaces that would be visible from the point of view (or “viewpoint”)of the image sensor. Processing of some or all of the remaining bricksmay be omitted. With such an approach, the selected bricks may be onesthat are likely to contain new information, which may be arrived at byculling bricks about which the output of the image sensor is unlikely toprovide useful information. The data output from the image sensor isunlikely to provide useful information about bricks that are eithercloser to or further from the image sensor than a surface indicated bythe depth map because those bricks are either empty space or behind asurface and therefore not depicted in images from the image sensor.

In some embodiments, one or more criteria may be applied to efficientlyselect a set of bricks for processing. An initial set of bricks may belimited to those within a frustum of an image sensor. A great number ofbricks outside the frustum may then be culled. More computer-resourceintense processing to update the 3D reconstruction may then be performedon the subset of bricks accepted for processing following the culling.Accordingly, a 3D representation of the scene to be updated is moreefficiently computed using processing on a reduced number of voxels.

An even greater reduction in processing may be achieved by cullingbricks based on the depth image. In accordance with some embodiments,culling and/or acceptance of bricks may be performed by projecting asilhouette of each brick in the initial set into the depth image. Suchculling may be based on whether the brick corresponds to a portion ofthe scene that the depth image indicates is in the vicinity of asurface. Bricks that can be simply identified as entirely in front of orentirely behind a surface may be culled, In some embodiments, such adetermination may be efficiently made. For example, a bounding boxaround the projection of the brick into the depth map may be used todetermine a maximum brick value and a minimum brick value along az-coordinate direction, which may be substantially perpendicular to a 2Dplane of the depth image. By comparing these maximum and minimum brickvalues to the distances represented by pixels in the depth map, bricksmay be culled and/or accepted for further processing. Such processingmay result in selection of bricks for initial processing that intersectwith surfaces, as reflected in the depth image, and/or that are in frontof a surface, as reflected in the depth image. In some embodiments, suchprocessing may differentiate between bricks that are in front of solidsurfaces and those that are in front of holey surfaces (i.e. bricksrepresenting regions for which the depth sensor was unable to reliablymeasure a distance to a surface).

In some embodiments, the culling/acceptance criteria may result inclassifying some or all of the bricks accepted for further processingsuch that processing algorithms for the computationally volumetricreconstruction may be tailored for the characteristics of the brick. Insome embodiments, different processing may be selected based on whetherthe brick is classified as intersecting a surface, being in front of asolid surface or being in front of a holey surface.

FIG. 9 shows a cross-sectional view of a scene 400 along a planeparallel to y-coordinate and z-coordinate. An XR system may representthe scene 400 by a grid of voxels 504. Conventional XR systems mayupdate each voxel of the grid of voxels based on every new depth imagecaptured by a sensor 406, which may be an image sensor or depth sensor,such that a 3D reconstruction generated from the grid of voxels canreflect changes in the scene. Updating in this fashion may consumesignificant computing resources and also cause artifacts at the outputof an XR system due to, for example, time latency caused by heavycomputing.

Described herein are techniques of providing accurate 3D reconstructeddata with low usage of computational resources, for example, by cullingparts of the grid of voxels 504 based at last in part on a camerafrustum 404 of an image sensor 406 and/or depth image captured by theimage sensor.

In the illustrated example, the image sensor 406 captures a depth image(not shown) including a surface 402 of the scene 400. The depth imagemay be stored in computer memory in any convenient way that capturesdistance between some reference point and surfaces in the scene 400. Insome embodiments, the depth image may be represented as values in aplane parallel to an x-axis and y-axis, as illustrated in FIG. 9, withthe reference point being the origin of the coordinate system. Locationsin the X-Y plane may correspond to directions relative to the referencepoint and values at those pixel locations may indicate distance from thereference point to the nearest surface in the direction indicated by thecoordinate in the plane. Such a depth image may include a grid of pixels(not shown) in the plane parallel to the x-axis and y-axis. Each pixelmay indicate a distance, in a particular direction, from the imagesensor 406 to the surface 402. In some embodiments, a depth sensor maybe unable to measure a distance to a surface in a particular direction.Such a condition, for example, may result if the surface is out of rangeof the image sensor 406. In some embodiments, the depth sensor may be anactive depth sensor, that measures distance based on reflected energy,but a surface may not reflect sufficient energy for an accuratemeasurement. Accordingly, in some embodiments, the depth image may have“holes,” where there are pixels to which no value is assigned.

In some embodiments, the reference point of the depth image may change.Such a configuration may allow the depth image to represent surfacesthroughout an entire 3D scene, not limited to the portions having apredetermined and limited range of angles with respect to a particularpoint of reference. In such embodiments, the depth image may indicatedistance to surfaces as the image sensor 406 moves through sixdegree-of-freedom (6DOF). In these embodiments, the depth image mayinclude a set of pixels for each of multiple reference points. In theseembodiments, a portion of the depth image may be selected based on a“camera pose,” representing the direction and/or orientation in whichthe image sensor 406 is pointing at the time image data is captured.

The image sensor 406 may have a field-of-view (FOV), which may berepresented by the camera frustum 404. In some embodiments, the infinitecamera frustum depicted may be reduced to a finite 3D trapezoidal prism408 by assuming a maximum depth 410 that the image sensor 406 canprovide, and/or a minimum depth 412 that the image sensor 406 canprovide. The 3D trapezoidal prism 408 may be a convex polyhedrondelimited by at six planes.

In some embodiments, one or more voxels 504 may be grouped into a brick502. FIG. 10A illustrates a portion 500 of the scene 400, which includeseight bricks 502. FIG. 10B illustrates an exemplary brick 502 comprising8³ voxels 504. Referring back to FIG. 9, the scene 400 may include oneor more bricks, sixteen of which are illustrated in the view depicted inFIG. 4. Each brick may be identifiable by a brick identity, for example,[0000]-[0015].

FIG. 11 depicts a volumetric representation hierarchy, which may beimplemented in some embodiments. In some embodiments, such volumetricrepresentation hierarchy may reduce latency for data transferring. Insome embodiments, a voxel grid of a physical world may be mapped toconform to a structure of a storage architecture for a processorcomputing AR content, such as the processor on which component 304executes. One or more voxels may be grouped into a “brick.” One or morebricks may be grouped into a “tile.” The size of a tile may correspondto a memory page of the storage medium that is local to the processor.Tiles may be moved between the local memory to remote memory, such asover a wireless connection, based on usage or anticipated usage inaccordance with a memory management algorithm.

In some embodiments, the uploading and/or downloading between theperception module 160 and the persisted world module 162 may beperformed on multiples of tiles in one operation. One or more tiles maybe grouped into a “RAM tile set.” The size of a RAM tile set maycorrespond to regions within a reconstruction range of a sensor worn bya user. One or more RAM tile sets may be grouped into a “global tileset.” The size of a global tile set may correspond to a perception rangeof a world reconstruction system (e.g., the perception range for thePerception module 160).

FIG. 12 is a flow chart, illustrating a method 600 of operating acomputing system to generate a 3D reconstruction of a scene, accordingto some embodiments. The method 600 may start with representing a scene(e.g., the scene 400) with one or more bricks (e.g., the brick 502),each brick including one or more voxels (e.g., the voxel 504). Eachbrick may represent a portion of the scene. The bricks may beidentifiable relative to a persistent coordinate system such that, evenas a pose of an image sensor (e.g., the image sensor 406) changes, thesame bricks represent the same volume in the scene.

At act 604, the method 600 may capture a depth image (e.g., the depthimage including the surface 402) from a depth sensor (e.g., the depthsensor 406). The depth sensor may be an active depth sensor,transmitting, for example, IR radiation and measuring time of flight fora reflection. Each such measurement represents distance to a surfacefrom the depth sensor in a particular direction. That depth informationmay represent the same volume as is represented by the bricks.

At act 606, the method 600 may cull a portion of the one or more bricksagainst a camera frustum (e.g., the finite 3D trapezoidal prism 408derived from the camera frustum 404) so as to produce a first one ormore bricks, which is a reduced set of bricks from the one or morebricks. Such culling may eliminate bricks representing portions of thescene outside the field of view of the image sensor at the time imagedata being processed was acquired. That image data, therefore, isunlikely to contain information useful in creating or updating a brick.

At act 608, the method 600 may cull a portion of the first one or morebricks against the depth image so as to produce a second one or morebricks, which is a reduced set of bricks from the first one or morebricks. At act 610, the method 600 may generate a 3D reconstruction ofthe scene based on the second one or more bricks.

Referring back to FIG. 9, given the surface 402 captured by the depthimage and the corresponding camera pose, voxels between the image sensor406 and the surface 402 may be empty. The farther away a voxel is behindthe surface 402 from the image sensor 406, it may be less certain thatthe voxel represents the inside of an object or empty space. The degreeof certainty may be represented by weight function, which weighs voxelupdates based on the distance to the surface 402. When a voxel lyingbehind the surface 402 (farther away from the image sensor 402) has aweight function above a threshold, the voxel may get no update or a zeroupdate (e.g., an update with zero changes). Also, all voxels not fallinginto the camera frustum 404 may not be updated or investigated for thisdepth image.

The method 600 may not only improve processing speed of volumetric depthimage fusion, but also consume less memory storage, which allows themethod 600 to run on wearable hardware. For example, a smallreconstruction volume of 5 m*5 m*3 m with 1 cm³ voxel size and 8 bytesper voxel (4 bytes for the distance value and 4 bytes for the weightvalue) would already require about 600 MB. The method 600 can categorizebricks by their distance to a surface with respect to a truncatedthreshold. For example, the method 600 can identify empty bricks (e.g.,the bricks that are culled, or the bricks that are away from the surfacebeyond the truncated threshold) so as to not allocate memory space forthe empty bricks. The method 600 can also identify bricks that are awayfrom the surface by the truncated threshold so as to store these bricksby a constant distance value of a negative truncation threshold andweight 1. The method 600 can also identify bricks that have a distanceto the surface between zero and the truncated threshold so as to storethese bricks with a constant SDF value of the positive truncationthreshold, but varying weight. Storing the distance or weight values,which are constant for a brick with a single value, may be anentropy-based compression for a zero-entropy field.

The method 600 may allow flagging a brick during the voxel update as“does not contain any part of the surface,” which can significantlyspeed up processing of a brick. That processing may include, forexample, converting the image of the portion of the scene represented bya brick into a mesh.

FIG. 13 shows an exemplary method 606 of culling a portion of the one ormore bricks against the camera frustum 404 of the image sensor 406,according to some embodiments. The method 606 may start with finding acubic axis-aligned bounding box (AABB) to contain the camera frustum404. The AABB may contains one or more bricks in a scene. The method 606may include dividing (act 704) the AABB into one or more sub-AABBs, andconducting (act 706) a camera frustum acceptance test. If the method 606determines at act 708 that a sub-AABB reaches the size of a brick, themethod 606 may produce (act 710) the first one or more bricks. If themethod 606 determines at act 708 that a sub-AABB is larger than the sizeof a brick, the method 606 may repeat acts 704-708 until a sub-AABBreaches the size of a brick.

For example, given the 3D trapezoidal prism 408 corresponding to thecamera frustum 404, an AABB may be found in constant time with a sidelength of a power of two that contains the 3D trapezoidal prism 408. TheAABB may be divided into eight sub-AABBs. Each of the eight sub-AABBsmay be tested for intersection with the camera frustum 404. When asub-AABB is determined as not intersecting the camera frustum 404,bricks corresponding this sub-AABB may be culled. A culled brick may berejected for further processing. When a sub-AABB is determined asintersecting the camera frustum 404, the sub-AABB may be further dividedinto eight sub-AABBs of the sub-AABB. Then each of the eight sub-AABBsof the sub-AABB may be tested for intersection with the camera frustum404. The iteration of dividing and testing continues until a sub-AABBcorresponds to a single brick. To determine whether a camera frustum 404intersects an AABB, a two-step test may be conducted. First, it may betested that whether at least one corner point of an AABB lies insideeach of the planes delimiting the camera frustum 404. Second, it may betested that whether every corner point of the camera frustum 404 liesinside the AABB such that some cases of AABBs, which do not intersectthe camera frustum 404, but are incorrectly classified as partiallyinside (e.g., having only one corner point on an edge of the frustum),may be captured.

A desirable byproduct of this two-step test is that it may be known foreach brick intersecting the camera frustum 404 whether it is completelyor only partially inside the camera frustum 404. For the brickscompletely inside the camera frustum 404, the test may be skipped forevery voxel whether it lies inside the camera frustum 404 later at theindividual voxel update.

FIG. 14 shows an exemplary method 706 of conducting a camera frustumacceptance test, according to some embodiments. The method 706 may startby testing (act 802) each of the one or more sub-AABBs against each ofplanes delimiting the camera frustum 404. At act 804, the method 706 maydetermine whether the tested sub-AABB is completely outside the camerafrustum 404. At act 806, if it is determined that the tested sub-AABB iscompletely outside the camera frustum 404, the method 706 may cull allthe bricks contained by this tested sub-AABB. At act 808, if it isdetermined that the tested sub-AABB is not completely outside the camerafrustum 404, the method 706 may determine whether the tested sub-AABB iscompletely inside the camera frustum 404.

At act 810, if it is determined that the tested sub-AABB is completelyinside the camera frustum 404, the method 706 may add all the brickscontained by the tested sub-AABB into the first one or more bricks. Atact 708, if it is determined that the tested sub-AABB is not completelyinside the camera frustum 404, which may indicate the tested sub-AABBintersects with the camera frustum 404, the method 706 may determinewhether the tested sub-AABB reaches the size of a brick.

At act 814, if it is determined that the tested sub-AABB equals to thesize of a brick, the method 706 may further determine whether everycorner point of the camera frustum 404 lies inside the tested sub-AABB.If it is determined that every corner point of the camera frustum 404lies inside the brick of the tested sub-AABB, the method 706 may cull(act 806) the brick of the tested sub-AABB. If it is determined that notevery corner point of the camera frustum lies inside the brick of thetested sub-AABB, the method 706 may add (act 810) the brick of thetested sub-AABB into the first one or more bricks.

FIG. 15 shows an exemplary method 608 of culling a portion of the firstone or more bricks against a depth image, according to some embodiments.The method 608 may start by conducting (act 902) a first depth imageacceptance test for each of the first one or more bricks. At act 904,the method 808 may determine whether a tested brick accepted by thefirst depth image acceptance test. If it is determined that the testedbrick is accepted by the first depth image acceptance test, which mayindicate the tested brick intersects with a surface in a scene, themethod 608 may apply (act 906) varying increment to selected voxels, andadd (act 914) the tested brick into the second one or more bricks.

At act 908, if it is determined that the tested brick is not accepted bythe first depth image acceptance test, the method 608 may conduct (act908) a second depth image acceptance test for the tested brick. At act910, it is determined whether the tested brick is accepted by the seconddepth image acceptance test. If it is determined that the tested brickis accepted by the second depth image acceptance test, which mayindicate the tested brick is in front of a solid or holey background inthe scene, the method 608 may apply (act 912) constant increment to allor selected voxels, and add (act 914) the tested brick into the secondone or more bricks. If it is determined that the tested brick is notaccepted by the second depth image acceptance test either, the method608 may cull (act 916) the tested brick.

FIG. 16 shows an exemplary method 902 of conducting a first depth imageacceptance test, according to some embodiments. For each brick to betested, the method 902 may start by determining (act 1002) a minimumbrick value (bmin) and a maximum brick value (bmax) along a directionparallel to the z-coordinate. The bmin value and bmax value may bepadded to account for an integration threshold, depth values beyondwhich indicate constant updates to voxels in a brick. At act 1004, themethod 902 may compute 2D pixel positions of corners of the tested brickby projecting the corners of the brick into the depth image. At act1006, the method 902 may compute a rectangle by building a convex hullof the 2D pixel positions of the corners of the brick. At act 1008, themethod 902 may test each pixel in the rectangle against the bmin valueand the bmax value. At act 1010, the method 902 may determine whetherall pixels in the rectangle have a depth value between the bmin valueand the bmax value. If it is determined that all the pixels in therectangle have a depth value between the bmin value and the bmax value,the method 902 may accept (act 1012) the brick. If it is determined thatnot all the pixels in the rectangle have a depth value between the bminvalue and the bmax value, the method 902 may conduct (act 908) a seconddepth image acceptance test for the brick.

FIG. 17 shows an exemplary method 908 of conducting a second depth imageacceptance test, according to some embodiments. The method 908 may startat act 1102 by categorizing all the pixels in the rectangle with respectto the bmin value and the bmax value. At act 1104, the method 908 maydetermine whether the tested brick is in front of a solid or holeybackground, for example, by using a table shown in FIG. 18. If it isdetermined that the tested brick is in front of a solid or holeybackground, the method 908 may accept (act 1106) the brick. If it isdetermined that the tested brick is not in front of a solid or holeybackground, at act 916 the method 908 may cull the brick.

FIGS. 19A-F depict an example of culling bricks representing a scene 190against a camera frustum 192. In FIG. 19A, the scene 190 is representedby a single AABB 194 a, which includes 16×16 bricks in the illustratedexample. In FIG. 19B, the single AABB 194 a is divided into foursub-AABBs 194 b, each of which includes 8×8 bricks. After conducting acamera frustum acceptance test (e.g., the method 706), one of the foursub-AABBs 194 b failed the camera frustum acceptance test and thus the8×8 bricks in the failed sub-AABB 194 b are culled and illustrated aswhite bricks. In FIG. 19C, each of the three sub-AABBs 194 that passedthe camera frustum acceptance test is further divided into four sub-AABB194 c, each of which includes 4×4 bricks. After conducting a camerafrustum acceptance test (e.g., method 706), eight of the sixteensub-AABBs 194 c failed the camera frustum acceptance test and thus thebricks in the failed sub-AABBs 194 c are culled and illustrated as whitebricks. Similarly, in FIG. 19D, a sub-AABB 194 d includes 2×2 bricks. InFIG. 19E, a sub-AABB 194 e includes a single brick and thus thesub-AABBs 194 e that passed the camera frustum test and respectivebricks are produced as the first plurality of bricks and illustrated asgrey bricks 196 f in FIG. 19F. In the illustrated example, if no cullingis performed, a world reconstruction component would compute all of the256 bricks. With the brick culling against the camera frustum, a worldreconstruction component only needs to compute the first plurality ofbricks, i.e. 34 bricks, and thus would render a result much faster.

FIG. 20A depicts an example of further culling the 34 bricks 196 fagainst a depth image including a surface 220 by conducting, forexample, the method 608. FIG. 20B depicts the result of culling againstthe depth image, showing that 12 bricks 222 a of the 34 bricks 196 fpassed the first depth image acceptance test (e.g., the method 904), 9bricks 222 b of the 34 bricks 196 f passed the second depth imageacceptance test (e.g., the method 910), and finally 13 bricks 222 c ofthe 34 bricks 196 f are culled after culling against the depth imagecomprising the surface 220. As a result, in the illustrated example,with the brick culling against the depth image, the number of brickscomputed by a world reconstruction component is further reduced to 21bricks. It also should be appreciated that computing speed of a wouldreconstruction component may be accelerated not only by the reducednumber of bricks but also by the classification of the bricks as aresult of the first and second depth image acceptance tests. Forexample, as discussed with respect to FIG. 15, a constant increment maybe applied to the 9 bricks 222 b that failed the first depth imageacceptance test but passed the second depth image acceptance test.Applying the constant increment in bulk may further improve computingspeed compared to applying varying increment to each voxel. Geometries(e.g., planes) in a scene may be obtained in XR systems to supportapplications, for example, a wall to place a virtual screen, and/or afloor to navigate a virtual robot. A common representation of a scene'sgeometry is a mesh, which may comprise groups of connected triangleshaving vertices and edges. Conventionally, a geometry in a scene isobtained by generating a mesh for the scene and searching the geometryin the mesh, which takes time to process, e.g., a few seconds, anddoesn't indicate relationships among geometries requested by differentqueries. For example, a first query may be for a table plane. Inresponse to the first query, a system may find a table plane and leave awatch on the table plane. Then a second query may be for a watch. Inresponse to the second query, a system may find all possible tableplanes and check whether there is a watch on the each table plane untilit finds the watch because there is no indication whether a table planeis the table plane from the response to the first query.

Described herein is a geometry extraction system. In some embodiments, ageometry extraction system may extract geometries while scanning a scenewith cameras and/or sensors, which allows a fast, efficient extractionthat can accommodate dynamic environment changes. In some embodiments, ageometry extraction system may persist extracted geometries in localand/or remote memories. The persisted geometries may be with uniqueidentifications such that different queries, for example, at differenttimestamps and/or from different applications, can share the persistedgeometries. In some embodiments, a geometry extraction system maysupport different representations of a geometry according to individualqueries. Planes are used as an exemplary geometry in the followingdescriptions of FIGS. 21-29. It should be appreciated that a geometryextraction system may detect other geometries to use in subsequentprocessing instead of or in addition to planes, including, for example,cylinders, cubes, lines, corners, or semantics such as glass surfaces orholes. In some embodiments, the principles described herein with respectto geometry extraction may be applicable to object extraction and thelike.

FIG. 21 shows a plane extraction system 1300, according to someembodiments. The plane extraction system 1300 may include depth fusion1304, which may receive multiple depth maps 1302. The multiple depthmaps 1302 may be created by one or more users wearing depth sensors,and/or downloaded from local/remote memories. The multiple depth maps1302 may represent multiple views of a same surface. There may bedifferences between the multiple depth maps, which may be reconciled bythe depth fusion 1304.

In some embodiments, the depth fusion 1304 may generate SDFs 1306 based,at least in part, on the method 600. Mesh bricks 1308 may be extractedfrom the SDFs 1306 by, for example, applying a marching cube algorithmover corresponding bricks (e.g., bricks [0000]-[0015] in FIG. 23). Planeextraction 1310 may detect planar surfaces in the mesh bricks 1308 andextract planes based at least in part on the mesh bricks 1308. The planeextraction 1310 may also extract surfels for each brick based, at leastin part, on corresponding mesh bricks. Surfels may include vertices in amesh but not the edges connecting neighboring vertices such that storingsurfels consumes less memory space than a mesh. A plane data store 1312may persist the extracted planes and surfels.

In some embodiments, XR applications may request and obtain planes fromthe plane data store 1312 by plane query 1314, which may be sent byapplication programming interfaces (APIs). For example, an applicationmay send to a plane extraction system 1300 information about itsposition, and ask for all planes near it (e.g., within five metersradius). The plane extraction system 1300 may then searches its planedata store 1312, and send the selected planes to the application. Theplane query 1314 may include information such as where the applicationneeds a plane, what kind of plane the application needs, and/or how theplane should look like (e.g., horizontal or vertical or angled, whichmay be determined by checking primitive normals of the planes in theplane data store).

FIG. 22 shows portions 1400 of the plane extraction system 1300,illustrating details on plane extraction 1310, according to someembodiments. The plane extraction 1310 may include dividing each of themesh bricks 1308 into sub-bricks 1402. Plane detection 1404 may beconducted for each of the sub-bricks 1402. For example, plane detection1404 may compare primitive normals of each mesh triangle in a sub-brick;merge those mesh triangles, with primitive normal differences smallerthan a predetermined threshold value, into one mesh triangle; andidentify a mesh triangle with an area larger than a predetermined areavalue as a plane.

FIG. 23 is a schematic diagram illustrating a scene 1500 represented bybricks [0000]-[0015] including voxels, and exemplary plane dataincluding a brick plane 1502, global plane 1504, and surfel 1506 in thescene, according to some embodiments. FIG. 23 illustrates a brick [0011]being divided into four sub-brick 1508. It should be appreciated that amesh brick may be divided into any suitable number of sub-bricks. Thegranularity of a plane detected by plane detection 1404 may bedetermined by the size of a sub-brick, while the size of a brick may bedetermined by the granularity of local/remote memories that stores thevolumetric 3D reconstruction data.

Referring back to FIG. 22, the plane detection 1404 may determine abrick plane (e.g., brick plane 1502) for each mesh brick based, at leastin part, on the detected planes for each sub-bricks in the mesh brick.The plane detection 1404 may also determine global planes that extendmore than one bricks (e.g., global plane 1504).

In some embodiments, the plane extraction 1310 may include plane update1406, which may update existing brick planes and/or global planes storedin the plane data store 1312 based, at least in part, on the planesdetected by plane detection 1404. The plane update 1406 may includeadding additional brick planes, removing some of the existing brickplanes, and/or replacing some of the existing brick planes with brickplanes detected by the plane detection 1404 and corresponding to thesame bricks such that real-time changes in the scene are persisted inthe plane data store 1312. The plane update 1406 may also includeaggregating a brick plane detected by the plane detection 1404 to anexisting global plane, for example, when a brick plane is detectedadjacent to an existing global plane.

In some embodiments, plane extraction 1310 may further include planemerging and splitting 1408. The plane merging may merge multiple globalplanes into one big global plane, for example, when a brick plane isadded and connects two global planes. Plane splitting may split oneglobal plane into multiple global planes, for example, when a brickplane in the middle of a global plane is removed.

FIG. 24 shows data structure in a plane data store 1312, according tosome embodiments. Global planes 1614 indexed by plane IDs 1612 may be atthe highest level of the data structure. Each global plane 1614 mayinclude multiple brick planes and surfels of bricks adjacent tocorresponding global planes, such that one brick plane may be persistedfor each brick while a global plane can be accurately presented whenedges of the global plane is not qualified as brick planes forcorresponding bricks. In some embodiments, surfels of bricks adjacent toa global plane are persisted instead of surfels of all bricks in a scenesince it would be sufficient for accurately presenting a global plane.For example, as illustrated in FIG. 23, the global plane 1504 extendsacross brick [0008]-[0010] and [0006]. The brick [0006] has the brickplane 1502, which is not part of the global plane 1504. With the datastructure in the plane data store 1312, when a plane query requests theglobal plane 1504, surfels of bricks [0006] and [0012] are checked todetermine whether the global plane 1504 extends into the bricks [0006]and [0012]. In the illustrated example, surfels 1506 indicates that theglobal plane 1504 extends into the brick [0006].

Referring back to FIG. 24, the global planes 1614 may bebi-directionally associated with corresponding brick planes 1610. Bricksmay be identifiable by brick IDs 1602. Bricks may be divided into planarbricks 1604, which include at least one plane, and non-planar bricks1606, which include no planes. Surfels for both planar bricks andnon-planar bricks may be persisted, depending on whether a brick isadjacent a global plane but not whether the brick includes planes. Itshould be appreciated that planes may be continuously persisted in theplane data store 1312 while an XR system is observing a scene,regardless whether or not there is a plane query 1314.

FIG. 25 shows planar geometry extraction 1702, which may extract planesfor an application's usage, when the application sends the plane query1314 to a plane data store 1312, according to some embodiments. Theplanar geometry extraction 1702 may be implemented as an API. The planequery 1314 may indicate requested planar geometry representation, forexample, outer rectangular planes, inner rectangular planes, or polygonplanes. According to the plane query 1314, plane search 1704 may searchand obtain plane data in the plane data store 1312.

In some embodiments, rasterization from plane coverage points 1706 maygenerate plane coverage points. An example is illustrated in FIG. 26A.There are four bricks [0000]-[0003], each has a brick plane 1802. Planecoverage points 1806 (or “rasterized points”) are generated byprojecting boundary points of the brick planes onto a global plane 1804.

Referring back to FIG. 25, rasterization from plane coverage points 1706may also generate a rasterized plane mask from the plane coveragepoints. According to the requested planar geometry representation by theplane query 1314, inner rectangular plane representation, outerrectangular plane representation, and polygon plane representation maybe extracted by inner rectangle extraction 1708, outer rectangleextraction 1710, and polygon extraction 1712, respectively. In someembodiments, an application can receive the requested planar geometryrepresentation within milliseconds from sending a plane query.

An exemplary rasterized plane mask 1814 is illustrated in FIG. 26B. Froma rasterized plane mask, various planar geometry representations may begenerated. In the illustrated example, a polygon 1812 is generated byconnecting some of the plane coverage points of the rasterized planemask such that none of the plane coverage pints in the mask is outsidethe polygon. An outer rectangle 1808 is generated such that the outerrectangle 1808 is the smallest rectangle surrounding the rasterizedplane mask 1814. Inner rectangles 1810 are generated by assigning “1”sto bricks with two plane coverage points and “0”s to bricks without twoplane coverage points to form a rasterized grid, determining groups ofbrick that are marked as “1” and aligned in a line parallel to an edgeof a brick (e.g., bricks [0001], [0005], [0009], and [00013] as a group,bricks [0013]-[0015] as a group), and generating one inner rectangle foreach determined group such that the inner rectangle is the smallestrectangle surrounding the respective group.

FIG. 27 shows a mesh for a scene 1900, according to some embodiments.FIGS. 28A-C show the scene 1900 represented by outer rectangular planes,inner rectangular planes, and polygon planes, respectively, according tosome embodiments.

FIG. 29 illustrated a less noisy 3D representation of the scene 1900,which is obtained by planarizing the mesh shown in FIG. 27 based onextracted plane data, for example, planes illustrated in FIGS. 28A-C.

Multi-Stage Block Mesh Simplification

In some embodiments, processing may be employed to reduce the complexityof a representation of an XR environment before that representation isstored or used in rendering functions, such as occlusion processing orcomputing physics of interactions among objects in the XR environment.For example, meshing component 160 d may simplify a mesh, or portion ofa mesh, before storing it in persisted world 162 as a mesh 162 c.

Such processing may entail performing operations in stages on arepresentation of an XR environment. Those stages may includesimplification operations that precede and follow a region-basedoperation. The region-based operation may, like the simplificationoperation, reduce the complexity of the representation of the XRenvironment. By staging the operations in this fashion, the totalprocessing to produce a simplified representation of the XR environmentmay be reduced while maintaining the quality of the representation ofthe XR environment. As a result, a simplified high-qualityrepresentation may be updated frequently such that an XR environment maybe frequently updated, improving the performance of the XR system, suchas by presenting a more realistic environment to a user.

The XR environment may represent a physical world and data representingthe XR environment may be captured by one or more sensors. However,techniques as described herein may be applied to XR environmentsregardless of the source of data representing the environment. In someembodiments, an XR environment may be represented by a mesh, includingone or more points and polygons, such as triangles, defined by subsetsof the points. A first simplification operation that precedes theregion-based operation may reduce the complexity of the representationof the environment. A mesh, for example, may be simplified by reducingthe number of such polygons in the mesh. As a specific example, thefirst simplification operation may employ a triangle reductionalgorithm, which may reduce the number of triangles used to representthe XR environment.

The region-based operation may be a shape detection operation that maydetect one or more shapes. A common shape detection operation is aplanarization operation in which the shapes that are detected areplanes. The detected planes may represent objects or portions ofobjects. Detection of planes may simplify processing to render an XRenvironment. A moving object being rendered in an XR environment, forexample, may move in a readily computable way upon colliding with aplane. Accordingly, identifying planes, may simplify subsequentrendering of moving objects in comparison to performing the computationbased on multiple polygons representing the same portion of theenvironment. Other shapes may be detected and used in subsequentprocessing instead of or in addition to planes, including cylinders,cubes, lines, corners, or semantics such as glass surfaces or holes.Such an operation may group polygons that represent a surface of adetected shape.

A second simplification operation that follows the region-basedoperation may further simplify the representation of the environment,such as by further reducing the number of polygons in therepresentation. The second simplification operation may focus onreducing the number of polygons within each region detected by theregion-based operation.

Such processing may enable a meshing service that processes sensor datacollected in a physical environment and provides a mesh to anapplication generating content. In some embodiments, the processing mayprovide simplified representations of virtual objects in a virtualenvironment.

In XR systems, such as virtual reality (VR), augmented reality (AR) andmixed reality (MR) systems, three-dimensional (3D) mesh data is oftenused for multiple purposes including, for example, occluding virtualcontent in a graphics/game engine based on physical objects in theenvironment, or computing the effect rigid body collisions for virtualobjects in a physics engine of a game engine. In some embodiments,requirements for the mesh may differ for different uses of the mesh, anda simplified mesh may be suitable for many such uses, with somesimplification techniques being better suited for some uses than others.

Accordingly, processing as described herein may be implemented with anyof multiple simplification techniques and/or simplification techniquesthat may be configured based on intended use of a simplified mesh. Theutility of a meshing service that provides a simplified mesh to multipleclient applications, which may use the mesh in different ways, may beimproved using processing as described herein. Each client applicationmay require a mesh with a different level of simplification. In someembodiments, an application accessing a meshing service may specify atarget simplification or the mesh to be provided to it. Meshsimplification methods as described herein may serve for multiple clientapplications including, for example, those that perform virtual contentocclusion, physics simulation, or environmental geometry visualization.The mesh processing described herein may have low latency and may beflexible in that it may optimize/bias the operation for different uses(e.g., flatten surfaces, varying triangle counts).

Mesh simplification methods as described herein may provide real-timeperformance (e.g., low latency to support on the fly (real-time)environment changes), local update capability (e.g., renewing parts of amesh that changed since last update), and planarized surfaces (e.g.,flattened planar surfaces to support robust physics simulation).

In some embodiments, a representation of an XR environment may besegmented into multiple blocks, some or all of which may be processed inparallel. In some embodiments, the resulting blocks may then berecombined. In some embodiments, the blocks may be defined with “skirts”that overlap adjacent blocks. The skirts enable recombination of theblocks with fewer and/or less noticeable discontinuities at theinterfaces of the reassembled blocks.

Accordingly, in some embodiments, a mesh simplification method mayinclude mesh block segmentation, pre-simplification, mesh planarization,and post-simplification. To speed up the process, a global mesh mayfirst be segmented into blocks of component meshes such that mesh blockscould be handled (e.g., processed) parallel. Then mesh blocks may beextended with skirts over boundaries between adjacent blocks. Withskirted mesh blocks, simplification may be conducted on individual meshblock, while the global mesh may be visually seamless althoughtopologically disconnected.

In some embodiments, a mesh simplification method may be suited for useby an application that uses the simplified mesh to representinteractions of objects in the XR environment, such as by making thesimplification processing plane-aware. To simplify a mesh, a three-stepsimplification process may be implemented. A mesh may be firstpre-simplified moderately using a relatively high target triangle count.Then plane areas may be detected by a region growing algorithm. The meshmay be planarized by projecting corresponding triangles to the detectedplanes. In some embodiments, the mesh may be regularized by adjustingplane (or primitive) normals to substantially vertical and parallel tothe detected planes. Afterwards, a post-simplification process may berun over the planarized mesh. The post-simplification processing mayfocus more on the detected plane areas, for example, simplify the meshof each detected plane area to reach a desired level of complexity(e.g., metric complexity), such as might be indicated by a target valueof one or more metrics.

FIG. 30 illustrates a method 3000 of generating a model of anenvironment represented by a mesh, according to some embodiments. Insome embodiments, the method 3000 may be performed on a meshing serviceon an XR platform. The method 3000 may start from an input meshrepresenting the environment at act 3002. In some embodiments, the inputmesh may have a high resolution, which may be indicated by the number oftriangles. The input mesh may be generated by a reconstruction system(e.g., a volumetric 3D reconstruction system) and the input mesh mayinclude 3D reconstruction data.

In some embodiments, the reconstruction system may generate a volumetric3D representation of the environment, which may create a data hierarchyof 3D information of the environment captured by one or more sensors.For example, a sensor may be a depth camera, which may capture 3Dinformation of the environment, for example, a stream of depth imageswith respective poses of the depth camera (i.e. camera poses). The 3Dinformation of the environment may be processed into a voxel grid. Eachvoxel may contain one or more signed distance functions (SDFs) thatdescribe whether the voxel lies inside or outside the geometries ofobjects in the environment. The voxels may be grouped into “bricks.”Each brick may include multiple voxels, for example, in cubic volumessuch as 8³ voxels. The bricks may be further grouped into “tiles.” Eachtile may include multiple bricks.

The size of a tile may be selected to facilitate memory operations in acomputing device. The size, for example, may be selected based on theamount of information about the environment that is maintained in activememory of a device that is processing such data. The system, forexample, may pass tiles between active memory, which is usually local tothe device, and other memory that has more latency, such as non-volatilememory or remote memory in a cloud. One or more whole or partial tilesmay contain information representing a “block” in a mesh or otherrepresentation of an environment.

In some embodiments, the volumetric 3D reconstruction system maygenerate the input mesh 3002 as a topologically-connected global mesh.In some embodiments, the volumetric 3D reconstruction system maygenerate the input mesh 3002 as a global mesh that is visually seamlessalthough topologically-disconnected. For example, atopologically-disconnected global mesh may be comprised of multiple meshblocks, each of which is generated from a block.

A reconstruction system may be configured to capture substantial detailof an environment, which enables the system to distinguish betweenadjacent portions of the representation that have relatively smalldifferences in their characteristics. Adjacent regions of differentproperties may be identified as different surfaces, resulting in thesystem identifying a large number of surfaces in the environment.However, such a system may capture details that are unnecessary for manyapplications, but are nonetheless processed. For example, areconstruction system may unnecessarily present bumps on a wall withmany triangles when two triangles making a rectangle would be asufficient representation of the wall for a client applicationrequesting meshes from the meshing service. In some embodiments, anapplication, when requesting a mesh from the meshing service, mayspecify a target simplification level of the requested mesh. That targetsimplification level may be expressed as a degree of compression, anumber of triangles per unit area, or in any other suitable way.

The method 3000 may efficiently generate a model of the environment,which is sufficient for the client application, from the input mesh. Atact 3004, the input mesh may be segmented into one or more first meshblocks, each of which may correspond to a block in the data hierarchy ofthe volumetric 3D representation of the environment.

Each first mesh block may represent a portion of the environment and mayhave a first value of a metric of complexity (e.g., mesh resolution). Insome embodiments, a metric of complexity of a mesh block indicates anumber of triangles in the mesh block. In some embodiments, processingmay be performed on mesh blocks sequentially and/or in parallel.However, simplification processing as described herein may be applied tothe entire mesh or any suitable portion (e.g., one or more mesh blocks).

Act 3006 represents a sub-process that is performed on each of multiplemesh blocks. The sub-processing may be performed on multiple mesh blocksindependently, such that the processing may be readily performed inparallel for some or all of the mesh blocks. The sub-process may beperformed on all mesh blocks or a subset of the mesh blocks selected forfurther processing. The subset of the mesh blocks may be selected based,at least in part, on a field-of-view of a device on which an applicationrequesting the simplified mesh is executing.

At act 3006, some of the first mesh blocks may be selected based on, forexample, objects described in the first mesh blocks or locations of thefirst mesh blocks. For each of the selected first mesh blocks, amulti-stage simplification may be performed. In some embodiments, themulti-stage simplifications on the selected first mesh blocks may beperformed in parallel and, as a result, the simplifications on theselected first mesh blocks may be completed approximately at a samepoint of time, though this may depend on the metric of complexity ofeach mesh block of the selected first mesh blocks.

The multi-stage simplification may include a pre-simplificationoperation, a region-based operation (e.g., a planarization operation),and a post-simplification operation. In some embodiments, themulti-stage simplification may be performed based on an input value fromthe client application. The input value may indicate required meshcomplexity by the client application (e.g., mesh resolution). The inputvalue from the client application may be the same or different for eachof the selected first mesh blocks.

At act 3012, a pre-simplification operation may be performed on aselected first mesh block to generate a second mesh block. Thepre-simplification operation may reduce the complexity of the block. Fora mesh block, the pre-simplification may reduce the number of polygonsin the mesh block. In some embodiments, the amount of pre-simplificationat act 3012 may be configurable. A target value, for example, may beprovided as an input to the processing at act 3012, for example, by theclient application. The target value may be a single value or multiplevalues of one or more specified or predetermined metrics. The one ormore metrics may include, for example, absolute triangle count, apercentage of the initial triangle counts, and/or a quadric errormetric, which may measure an average squared distance between asimplified mesh and an original mesh (e.g., the input mesh 3002).

The target value may be provided in any suitable way. An instance of themethod 3000, for example, may be pre-configured with a target value. Insome embodiments, the target value may be supplied through an API by anapplication requesting a mesh from the meshing service performing themethod 3000. For example, the target value for act 3012 may be a finaltarget requested by a rendering function (e.g., the requestingapplication). In some embodiments, the target value provided as an inputmay be adjusted or overridden to ensure that sufficient data remains inthe mesh for subsequent processing. For example, processing in act 3014may require a minimum number of triangles and a target value provided byan application may be replaced by that minimum value if the target valueis below the minimum number of triangles.

In such embodiments, the pre-simplified mesh may have values of one ormore metrics such that the pre-simplified mesh can be processed fasterduring the region-based operation than the original block segmentedinput mesh, while still containing all or most of the regions of theoriginal block segmented input mesh.

Without controlling the values of the one or more metrics, thesimplified mesh may be too coarse, unevenly distributed, and/or losemany regions of the original block segmented input mesh that arerequired in the following region-based operation.

The second mesh block produced in act 3012 may have a second value ofthe metric of complexity, which may be less than the first value of themetric complexity. In some embodiments, the pre-simplification operationof act 3012 may be performed using a triangle reduction algorithm.

At act 3014, a shape detection operation may be performed on the secondmesh block to generate a third mesh block. A planarization operation isillustrated as an example. The third mesh block may have a third valueof the metric of complexity. In some embodiments, the third value ofmetric complexity may be the same as the second value of the metriccomplexity. In some embodiments, the third value of metric complexitymay be less than the second value of metric complexity. Theplanarization operation may include detecting planar areas in the secondmesh block, for example, using a region growing algorithm, projectingmeshes of the detected planar areas to corresponding planes, adjustingplane normals of the detected planar areas to be substantiallyperpendicular to the corresponding planes, and simplifying the projectedmeshes on each of the corresponding planes based on, for example, atarget triangle count. In some embodiments, plane normals of thedetected planar areas may be adjusted before projecting meshes of thedetected planar areas to the corresponding planes.

At act 3016, a post-simplification operation may be performed on thethird mesh block to generate a fourth mesh block. In some embodiments,processing at act 3014 may desirably be performed on a mesh with moreresolution than is required in the simplified mesh that is to be outputfrom method 3000. In some embodiments, processing at act 3016 maysimplify the entire mesh block to reach a desired level of complexity(e.g., metric complexity), such as might be indicated by a target valueof one or more metrics, which may be the same or different target thanwas supplied to act 3012. In some embodiments, the post-simplificationoperation at act 3016 may focus on reducing the number of polygonswithin each plane detected by the planarization operation at act 3014.

The fourth mesh block may have a fourth value of the metric ofcomplexity, which may be less than the third value of the metriccomplexity. In some embodiments, a percentage reduction between thethird value of the metric complexity and the fourth value of the metriccomplexity may be greater than a percentage reduction between the firstvalue of the metric complexity and the second value of the metriccomplexity. In some embodiments, the percentage reduction between thethird value of the metric complexity and the fourth value of the metriccomplexity may be at least two times greater than the percentagereduction between the first value of the metric complexity and thesecond value of the metric complexity. In some embodiments, thepost-simplification operation of act 3016 may be performed using atriangle reduction algorithm. In some embodiments, thepost-simplification operation of act 3016 may be performed using thesame simplification algorithm as the pre-simplification operation of act3012.

At act 3008, the simplified selected blocks may be combined with otherselected mesh blocks similarly processed as in act 3006 and/or may becombined with unselected blocks into a new mesh of the environment. Atact 3010, the new mesh of the environment may be provided to a clientapplication. In some embodiments, the new mesh of the environment may bereferred to as a simplified mesh.

In some embodiments, act 3008 may be skipped. The simplified mesh blocksmay be sent directly to a client application, in which the mesh blocksmay be visually seamless although topographically disconnected.

FIG. 31 illustrates an example of segmenting a mesh representation 3100of an environment into mesh blocks, according to some embodiments. Themesh representation 3100 may be segmented into four mesh blocks: meshblocks A-D. In some embodiments, mesh blocks may correspond to regionswith the same volume in a physical world that belongs to theenvironment. In some embodiments, mesh blocks may correspond to regionswith different volumes in the physical world that belongs to theenvironment. For example, when the physical world is an office room, theoffice room may be divided into regions, each of which may be one cubicfoot. One block may include a 3D representation of one region of theoffice room.

Although the mesh representation 3100 of an environment is illustratedin two dimensions (2D), it should be appreciated that an environment maybe three-dimensional, correspondingly, represented by a 3D meshrepresentation. Although the mesh representation 3100 of an environmentis illustrated as a combination of four mesh blocks, it should beappreciated that an environment may be represented by any suitablenumber of mesh blocks, for example, two, three, five, six or more.

Representation 3100 may be segmented into four parts: parts 3102, 3104,3106, and 3108, for example, illustrated by solid lines 3110. In someembodiments, parts 3102, 3104, 3106, and 3108 may be designated as meshblocks A-D respectively.

When a mesh block is updated, it may continue to interface with anadjacent block that was not updated. As a result, discontinuities at theboundaries between the mesh blocks may arise. If regions represented bythe adjacent blocks have discontinuities, the fused mesh may beinterpreted, in subsequent processing, as indicating that there arecracks between the adjacent blocks. In some embodiments, such cracks inthe representation of the physical world space may be interpreted asspace with infinite depth. As such, a space may be an artifact of therepresentation of the physical world, rather than actually a feature.Any application using such a fused mesh to generate a representation ofobjects in the physical world may not generate output correctly. Forexample, an application rendering a virtual character on surfaces withinthe physical world may render that character to appear as if it fellthrough the crack, which would not create the desired appearance of theobject.

To reduce the appearance of such cracks, in some embodiments, a portionof adjacent blocks may represent a same region of the physical world.For example, interface regions between the adjacent blocks may berepresented by a portion of each of the adjacent blocks, which mayenable easy independent updating and/or rendering that accounts forlevel of detail (LOD) (e.g., decreasing the complexity of a 3Dreconstruction of a portion of the physical world as the portion movesout the user's field of view). Even if one block is updated, and itsadjacent block is not, the fused mesh may be represented by combiningdata representing the interface region from both block. As a specificexample, when fusing the updated block with the adjacent blocks, aphysics engine may determine an overlapping region of the adjacentblocks based on, for example, which one of the adjacent blocks should beobservable in their overlapping region. A block-based data structure maybe adopted with skirts, zippers, or any other suitable methods torepresent interface regions between adjacent blocks so that when a blockis updated, it will continue to interface with adjacent blocks that werenot updated. Such an approach may have the appearance of “papering over”cracks between adjacent blocks. Therefore, blocks may be updatedindependently of adjacent blocks.

In the illustrated example in FIG. 31, regions at the boundaries ofparts 3102, 3104, 3106, and 3108 may be designated as skirts,illustrated by dashed lines 3112. In some embodiments, each of meshblocks A-D may include one of parts 3102, 3104, 3106, and 3108, and acorresponding skirt. For example, mesh block B may include the part 3104and a skirt 3114 that overlaps with boundary parts of mesh block B'sneighboring mesh blocks A, C, and D such that cracks between the meshblocks may be papered over when the blocks are connected into a singlemesh. Mesh blocks A, C, and D may also include corresponding skirts.Accordingly, before returning a single connected 3D mesh representationto an application, a processor may paper over any cracks between themesh blocks.

In some embodiments, block meshes that include skirts may be sentdirectly to an application, without being combined to atopographically-connected global mesh. The application may have a globalmesh comprised of block meshes, which is visually seamless althoughtopologically disconnected.

FIGS. 32A-32D illustrate a mesh evolution of an exemplary mesh block3201 during a multi-stage simplification, according to some embodiments.Mesh block 3201 may include vertices 3206, edges 3208, and faces 3210.Each face may has a normal, which may be represented by multiplecoordinates, shown for example as x, y, z in FIG. 32A.

A pre-simplification operation may be performed on mesh block 3201 togenerate mesh block 3202. An edge collapse transformation may be used.In the illustrated example, mesh block 3202 reduces the number of facesof mesh block 3201 from ten to eight. The resulting faces of mesh block3202 may each have a set of corresponding normals (e.g., x1, y1, z1; x2,y2, z2; . . . x8, y8, z8).

A planarization operation may be performed on mesh block 3202 togenerate mesh block 3203. The planarization operation may includedetecting planar areas in mesh 3202 based on, for example, the plane (orprimitive) normals of the faces. Values of plane normals x1, y1, z1 of afirst face 3212 and plane normals x2, y2, z2 of a second face 3214 maybe compared. The comparison result of the plane normals of the first andsecond faces may indicate angles between the plane normals (e.g., anglesbetween x1 and x2). When the comparison result is within a thresholdvalue, it may be determined that the first and second planes are on asame planar area. In the illustrated example, planes 3212, 3214, 3216,and 3218 may be determined as on a first planar area corresponding toplane 3228; planes 3220, 3222, 3224, and 3226 may be determined as on asecond same planar area corresponding to plane 3230.

The planarization operation may also include projecting triangles formedby edges of planes 3212, 3214, 3216, and 3218 to plane 3228 asillustrated by dashed lines 3232, and projecting triangles formed byedges of planes 3220, 3222, 3224, and 3226 to plane 3230 as illustratedby dashed lines 3234. The planarization operation may also includeadjusting plane normals for planes 3212, 3214, 3216, and 3218 to be thesame as plane normals for plane 3228 (x_a, y_a, z_a), and adjustingplane normals for planes 3220, 3222, 3224, and 3226 to be the same asplane normals for plane 3230 (x_b, y_b, z_b).

A post-simplification operation may be performed on mesh block 3203 togenerate mesh block 3204. In the illustrated example, mesh block 3204reduces the number of faces of mesh block 3203 from eight to four.

FIGS. 33A and 33B-36A and 36B illustrate the effect of simplification,showing side-by-side the same portion of a physical world with andwithout a simplification applied. These figures provide a graphicalillustration that simplification is able to provide usable informationto operate an AR system, while providing less data that needs to beprocessed.

FIGS. 33A and 33B show representations of the same environment withoutsimplification and with simplification through triangle reduction,respectively. Such processing is an example of processing that may beperformed at the pre-simplification and post-simplification blocks 3012and 3016 in FIG. 30.

FIGS. 34A and 34B show close-up representations of the same environmentwithout simplification through triangle reduction and withsimplification, respectively.

35A and 35B show representations of the same environment withoutplanarization and with planarization, respectively. Such processing isan example of processing that may be performed at the planarizationblock 3014 in FIG. 30.

36A and 36B show representations of the same environment withoutsimplification and with simplification through removal of disconnectedcomponents, respectively. Such processing is an example of analternative embodiment of region-based operation that may be performedat block 3014 in FIG. 30.

Caching and Updating of Dense 3D Reconstruction Data

In some embodiments, 3D reconstruction data may be captured, persistedand updated in blocks, which may allow local update while maintainingneighbor consistency. The block-based 3D reconstruction datarepresentation may be used in connection with a multi-layer cachingmechanism that retrieves, prefetches, and stores 3D data efficiently forAR and MR applications, including both single-device and multi-deviceapplications. For example, volumetric information 162 a and/or meshes162 c (FIG. 6) may be stored in blocks. Use components 164 may receiveinformation about the physical world using this block-basedrepresentation. Similarly, perception component 160 may store andretrieve such information in blocks.

These techniques expand the capabilities of portable devices, withlimited computational resources, to present AR and MR content with highrealism. Such techniques, for example, may be used to efficiently updateand manage the output of real-time or offline reconstruction andscanning in a mobile device having limited resource and connection tothe Internet (continuous or noncontinuous). The techniques makeavailable to a wide variety of mobile AR and MR applications up-to-date,accurate, and comprehensive 3D reconstruction data, in eithersingle-device applications or multi-device applications sharing andupdating the same 3D reconstruction data. These 3D reconstruction datamay be in any suitable formats including meshes, point clouds, voxels,and the like.

Some AR and MR systems have attempted to simplify presentation of MR andAR scenes by limiting the amount of 3D reconstruction data beingprocessed at any time. Sensors used to capture 3D information may have amaximum reconstruction range that may limit a bounding volume around afield of view of a sensor. To reduce the amount of 3D reconstructiondata, some reconstruction systems only hold areas near the field of viewof the sensors in an active working memory and store other data insecondary storage. For example, areas near the field of view of thesensors are stored in CPU memory while other data are persisted to alocal cache (e.g., disk) or over a network to remote storage (e.g. inthe cloud).

The computational cost of generating the information stored in CPUmemory, even though limited, may still relatively high. Some AR and MRsystems continually recompute a global representation of the environmentthat the reconstruction system has ever seen in order to selectinformation to store in CPU memory, which may be prohibitively expensivefor interactive applications. Other AR and MR systems compute just alocal update to a connected representation using some approaches may besimilarly expensive, particularly for simplified mesh, as it requiresbreaking apart the existing mesh, calculating another mesh with theidentical boundaries, and re-connecting the mesh parts.

In some embodiments, 3D reconstruction data may be segmented intoblocks. The 3D reconstruction data may be transmitted among storagemediums on the basis of blocks. For example, a block may be paged out ofan active memory and persisted to a local or remote cache. The systemmay implement a paging algorithm in which active memory associated witha wearable device (e.g., head-mounted display device) stores blocksrepresentative of a portion of a 3D reconstruction of the physical worldin a field of view of a user of the wearable device. The wearable devicemay capture data about portions of the physical world commensurate withthe field of view of the user of the wearable device. As the physicalworld changes in the field of view of the user, blocks representing thatregion of the physical world may be in an active memory from which theycan be readily updated. As the user's field of view changes, blocksrepresenting regions of the physical world that move out of the user'sfield of view may be moved to a cache such that blocks representingregions of the physical world coming into the user's field of view maybe loaded into an active memory.

In some embodiments, a coordinate frame may be created for a portion ofa physical world that is to be 3D reconstructed. Each block in a 3Drepresentation of the portion of the physical world may correspond todifferent regions of a physical world that is identifiable using thecoordinate frame.

In some embodiments, when a block is updated, the updated block maycontinue to interface with adjacent blocks that may not have beenupdated. If regions represented by the adjacent blocks do not overlap,there may be cracks in the fused mesh of the adjacent blocks. In someembodiments, such cracks in the representation of the physical worldspace may be interpreted as space with infinite depth. As such, a spacemay be an artifact of the representation of the physical world, ratherthan actually a feature. Any application using such a fused mesh togenerate a representation of objects in the physical world may notgenerate output correctly. For example, an application rendering avirtual character on surfaces within the physical world may render thatcharacter to appear as if it fell through the crack, which would notcreate the desired appearance of the object. Therefore, in someembodiments, a portion of adjacent blocks may represent a same region ofthe physical world, for example, interfaces between the adjacent blocks,which may enable easy independent updating and/or rendering thataccounts for level of detail (LOD) (e.g., decreasing the complexity of a3D reconstruction of a portion of the physical world as the portionmoves out the user's field of view). For example, when a block isupdated, its adjacent blocks may not be updated. When fusing the updatedblock with the adjacent blocks, a physics engine may determine anoverlapping region of the adjacent blocks based on, for example, whichone of the adjacent blocks should be observable in their overlappingregion. A block-based data structure may be adopted with skirts,zippers, or any other suitable methods to represent interfaces betweenadjacent blocks so that when a block is updated, it will continue tointerface with adjacent blocks that were not updated. Such an approachmay have the appearance of “papering over” cracks between adjacentblocks. Therefore, blocks that have changed may be updated independentlyof adjacent blocks.

In some embodiments, these techniques may be used in an AR and/or a MR“platform,” which receives and processes data from sensors worn by oneor more users. This sensor data may be used to create and update 3Dreconstruction data, representative of portions of the physical worldencountered by the user(s). A reconstruction service may continuallyreconstruct a 3D representation of the physical world when the sensorsare capturing and updating data. One or more techniques may be used todetermine the blocks impacted by changes in the physical world and thoseblocks may be updated. This 3D reconstruction data may then be providedto applications that use the 3D reconstruction data to render scenes todepict virtual reality objects positioned in or interacting with objectsin the physical world. This data may be provided to applications throughan application programming interface (API). The API may be a push orpull interface, either pushing data to the application when relevantportions change or responding to a request from the application forup-to-date information.

In the example of a pull interface, when applications request 3Dreconstruction data of the physical world, a reconstruction service maydetermine an appropriate version of each block that needs to be providedto the application, enabling the reconstruction service to start withthe most up-to-date blocks. The reconstruction service, for example, maysearch for blocks that were previously persisted. A single device systemmay enable a single device to contribute 3D reconstruction data aboutthe physical world. In the single device system, if the region of thephysical world requested is or extends outside an active area (e.g., anarea in a current field of view of the device), persisted blocks may beused directly as the most up-to-date blocks, because the persistedblocks would not have been updated since they were persisted when thatarea moved out a field of view of the device. On the other hand, amulti-device system may enable multiple devices to contribute 3Dreconstruction data about the physical world, such as with cloudpersistence or peer-to-peer local cache sharing of blocks. Each devicemay update a region within its active area, which may be persisted. Amulti-device system may create a coordinate frame such that blocksgenerated by different devices are identifiable using the coordinateframe. Therefore, blocks requested by the applications generating datafor a first device may be based on updates from other devices, if thoseupdates were made after any version made by the first device. Blocksbuilt using data from first device and other devices may be merged byusing the coordinate frame.

The selected blocks may be used to provide 3D reconstruction data aboutthe physical world in any suitable format, but a mesh is used as anexample of a suitable representation herein. A mesh may be created byprocessing image data to identify points of interest in the environment,such as edges of objects. These points may be connected, forming themesh. Groups of points, conventionally three points, in the mesh thatare associated with the same object, or portion thereof, define surfacesof the object, or portions thereof. Information stored in conjunctionwith the groups of points describe the surfaces in the environment. Thisinformation may then be used in varied ways to render and/or displayinformation with respect to the environment. The selected blocks may beused to provide 3D reconstruction data in any suitable way. In someembodiments, the most up-to-date block may be supplied. In someembodiments, the most up-to-date block may be used to determine whetheran update of the block is required.

For example, in some embodiments, in a multi-device system, when blocksrequested by an application have been identified, a reconstructionservice may check blocks persisted by other devices to determine ifthere have been any significant updates (e.g., via geometry changemagnitude thresholds or timestamps), re-run meshing on the blocks thathave changed, and then persist these updated mesh blocks.

In some embodiments, when a set of blocks requested by an applicationhas been identified, if the application requested a connected mesh, theset of blocks may be processed into a global mesh, which may be may betopologically-connected, or visually seamless althoughtopologically-disconnected using any suitable techniques such as skirtsand zippers.

In some embodiments, an application (e.g., graphics/game engine) mayupdate its internal block (e.g., block stored in active memory and/orlocal cache) when a block change occurs. A reconstruction service mayknow which blocks the application has, and can therefore calculate whichother (e.g., neighboring) blocks need to be updated in the engine tomaintain correct overlap with the skirt/zipper as the in-view blocksupdate.

In some embodiments, an AR and/or MR platform may be implemented tosupport execution of AR and/or MR applications, for example, on a mobiledevice. An application executing on or generating data for presentationthrough a user interface may request 3D reconstruction datarepresentative of the physical world. That 3D reconstruction data may besupplied from active memory on the device, which may be updated with 3Dreconstruction data representative of the physical world in a field ofview of the device as the user changes their field of view. The 3Dreconstruction data in active memory may represent an active area forthe mobile device. In some embodiments, 3D reconstruction data outsidean active area of the device may be stored in other memory, such as in alocal cache on the device or coupled to the device with a low latencyconnection. In some embodiments, 3D reconstruction data outside anactive area of the device may also be stored in a remote cache, such asin the cloud, which is accessible to the device over a higher latencyconnection. As the user changes their field of view, the platform mayaccess/load 3D reconstruction data from the cache to add to the activememory to represent regions that move into the user's field of view. Theplatform may move other data, representing regions that move out of theuser's field of view, to cache.

Prediction of the device's movements, which may result in regions movinginto the device's field of view, and other regions moving out of thedevice's field of view, may be used to initiate transfer of 3Dreconstruction data between the active memory and cache. Prediction ofmovement may, for example, be used to select 3D reconstruction data fortransfer into and/or out of the active memory. In some embodiments,predicted movement may be used to transfer 3D reconstruction data intoand/or out of the local cache by retrieving from or transferring 3Dreconstruction data to the remote cache. Exchanging 3D reconstructiondata between the local cache and remote cache based on a user'spredicted movements may ensure that 3D reconstruction data is availablewith low latency to move into the active memory.

In embodiments in which regions of the physical world are representedwith blocks, initiating transfer of blocks may entail pre-requestingblocks representing areas predicted to come into the user's field ofview. For example, if the platform, based on sensor or other data,determines that the user is walking in a particular direction at aparticular speed, it may identify regions that are likely to come intothe user's field of view and transfer blocks representing these areas tothe local cache on the mobile device. If the mobile device is a wearabledevice, such as a set of glasses, predicting motion may entail receivingsensor data indicating location, direction, and/or rotation of theuser's head.

FIG. 37 illustrates a system 3700 that enables interactive X realityenvironments for multiple users, according to some embodiments. Thesystem 3700 may include a computing network 3705 included of one or morecomputer servers 3710 connected through one or more high bandwidthinterfaces 3715. The servers in the computing network need not beco-located. The one or more servers 3710 may each include one or moreprocessors for executing program instructions. The servers also includememory for storing the program instructions and data that is used and/orgenerated by processes being carried out by the servers under directionof the program instructions. The system 3700 may include one or moredevices 3720 including, for example, AR display systems 80 (e.g.,viewing optics assembly 48 in FIG. 3B).

The computing network 3705 communicates data between the servers 3710and between the servers and the devices 3720 over one or more datanetwork connections 3730. Examples of such data networks include,without limitation, any and all types of public and private datanetworks, both mobile and wired, including for example theinterconnection of many of such networks commonly referred to as theInternet. No particular media, topology or protocol is intended to beimplied by the figure.

In some embodiments, the devices may be configured for communicatingdirectly with computing network 3705, or any of the servers 3710. Insome embodiments, the devices 3720 may communicate with the remoteservers 3710 and, optionally, with other devices and AR display systemslocally, through a local gateway 3740 for processing data and/or forcommunicating data between the network 3705 and one or more devices3720.

As illustrated, gateway 3740 is implemented as a separate hardwarecomponent, which includes a processor for executing softwareinstructions and memory for storing software instructions and data. Thegateway has its own wired and/or wireless connection to data networksfor communicating with the servers 3710 comprising computing network3705. In some embodiments, gateway 3740 can be integrated with thedevice 3720, which is worn or carried by a user. For example, thegateway 3740 may be implemented as a downloadable software applicationinstalled and running on a processor included in the device 3720. Thegateway 3740 provides, in one embodiment, one or more users access tothe computing network 3705 via the data network 3730. In someembodiments, gateway 3740 may include communication links 76 and 78.

Servers 3710 each include, for example, working memory and storage forstoring data and software programs, microprocessors for executingprogram instructions, graphics processors and other special processorsfor rendering and generating graphics, images, video, audio andmulti-media files. Computing network 3705 may also include devices forstoring data that is accessed, used or created by the servers 3710. Insome embodiments, computing network 3705 may include remote processingmodule 72 and remote data repository 74.

Software programs running on the servers and optionally the devices 3720and gateways 3740, are used to generate digital worlds (also referred toherein as virtual worlds) with which users interact with the devices3720. A digital world is represented by data and processes that describeand/or define virtual, non-existent entities, environments, andconditions that can be presented to a user through the device 3720 forusers to experience and interact with. For example, some type of object,entity or item that will appear to be physically present wheninstantiated in a scene being viewed or experienced by a user mayinclude a description of its appearance, its behavior, how a user ispermitted to interact with it, and other characteristics. Data used tocreate an environment of a virtual world (including virtual objects) mayinclude, for example, atmospheric data, terrain data, weather data,temperature data, location data, and other data used to define and/ordescribe a virtual environment. Additionally, data defining variousconditions that govern the operation of a virtual world may include, forexample, laws of physics, time, spatial relationships and other datathat may be used to define and/or create various conditions that governthe operation of a virtual world (including virtual objects).

The entity, object, condition, characteristic, behavior or other featureof a digital world will be generically referred to herein, unless thecontext indicates otherwise, as an object (e.g., digital object, virtualobject, rendered physical object, etc.). Objects may be any type ofanimate or inanimate object, including but not limited to, buildings,plants, vehicles, people, animals, creatures, machines, data, video,text, pictures, and other users. Objects may also be defined in adigital world for storing information about items, behaviors, orconditions actually present in the physical world. The data thatdescribes or defines the entity, object or item, or that stores itscurrent state, is generally referred to herein as object data. This datais processed by the servers 3710 or, depending on the implementation, bya gateway 3740 or the device 3720, to instantiate an instance of theobject and render the object in an appropriate manner for the user toexperience through the device 3720.

Programmers who develop and/or curate a digital world create or defineobjects, and the conditions under which they are instantiated. However,a digital world can allow for others to create or modify objects. Oncean object is instantiated, the state of the object may be permitted tobe altered, controlled or manipulated by one or more users experiencinga digital world.

For example, in one embodiment, development, production, andadministration of a digital world is generally provided by one or moresystem administrative programmers. In some embodiments, this may includedevelopment, design, and/or execution of story lines, themes, and eventsin the digital worlds as well as distribution of narratives throughvarious forms of events and media such as, for example, film, digital,network, mobile, augmented reality, and live entertainment. The systemadministrative programmers may also handle technical administration,moderation, and curation of the digital worlds and user communitiesassociated therewith, as well as other tasks typically performed bynetwork administrative personnel.

Users interact with one or more digital worlds using some type of alocal computing device, which is generally designated as the device3720. Examples of such devices include, but are not limited to, asmartphone, tablet device, heads-up display (HUD), gaming console, orany other device capable of communicating data and providing aninterface or display to the user, as well as combinations of suchdevices. In some embodiments, the device 3720 may include, orcommunicate with, local peripheral or input/output components such as,for example, a keyboard, mouse, joystick, gaming controller, hapticinterface device, motion capture controller, audio equipment, voiceequipment, projector system, 3D display, and holographic 3D contactlens.

FIG. 38 is a schematic diagram, illustrating an electronic system 3800,according to some embodiments. In some embodiments, the system 3800 maybe a part of the system 3700 of FIG. 37. The system 3800 may include afirst device 3810 (e.g., a first portable device of a first user) and asecond device 3820 (e.g., a second portable device of a second user).The devices 3810 and 3820, for example, may be the devices 3720 and ARdisplay system 80 of FIG. 37. The devices 3810 and 3820 may communicatewith a cloud cache 3802 through networks 3804 a and 3804 b,respectively. In some embodiments, the cloud cache 3802 may beimplemented in a memory of one or more servers 3710 of FIG. 37. Thenetworks 3804 a and 3804 b may be an example of data network 3730 and/orlocal gateway 3740 of FIG. 37.

The devices 3810 and 3820 may be individual AR systems (e.g., thedevices 3720). In some embodiments, the devices 3810 and 3820 mayinclude AR display systems worn by their respective users. In someembodiments, one of the devices 3810 and 3820 may be an AR displaysystem worn by a user; the other may be a smartphone held by a user.Although two devices 3810 and 3820 are illustrated in the example, itshould be appreciated that system 3800 may include one or more devices,and the one or more devices may be operating the same type of AR systemsor different types of AR systems.

The devices 3810 and 3820 may be portable computing devices. The firstdevice 3810, for example, may include a processor 3812, a local cache3814, and one or more AR applications 3816. The processor 3812 mayinclude a computing portion 3812 a configured to execute computerexecutable instructions to provide a 3D representation (e.g., 3Dreconstruction data) of a portion of a physical world based at least inpart on data collected with one or more sensors (e.g., depth sensor 51,world camera 52, and/or inertial measurement units 57 of FIG. 3B).

The computing portion 3812 a may represent the physical world as one ormore blocks. Each block may represent objects in a different region ofthe physical world. Each region may have a corresponding volume. In someembodiments, the blocks may represent regions with a same volume. Insome embodiments, the blocks may represent regions with differentvolumes. For example, when the physical world is an office room, theoffice room may be divided into cubes, each of which may be one cubicfoot. One block may include a 3D representation (e.g., 3D reconstructiondata) of one cube of the office room. In some embodiments, the officeroom may be divided into regions with various volumes, and each volumemay include a similar amount of 3D information (e.g., 3D reconstructiondata) such that the data size of a 3D representation of each region maybe similar. That representation may be formatted in such a way as tofacilitate further processing, such as occlusion processing, todetermine whether a virtual object is occluded by a physical object orphysics processing, to determine how a virtual object should move ordeform when interacting with physical objects in the physical world. Theblocks, for example, may be formatted as mesh blocks, in which featuresof objects in the physical world, such as corners, become points in themesh block, or are used as points to create a mesh block. Connectionsbetween points in the mesh may indicate groups of points on the samesurface of a physical object.

Each block may have one or more versions, with each version containingdata (e.g., volumetric 3D reconstruction data such as voxels, and/ormesh that may represent surfaces in a region represented by a respectiveblock) representing its corresponding region based on data from a pointin time. The computing portion 3812 a may create new versions of a blockas additional data becomes available, such as data indicating thatobjects in the physical world have changed or additional data from whicha more accurate representation of the physical world may be created.Additional data may come from sensors on the device (e.g., the device3810 and/or 3820). In some embodiments, additional data may come fromremote sensors, and may be obtained, for example, over a networkconnection.

The processor 3812 may also include an active memory 3812 b, which maybe configured to store blocks that are in a device's field of view. Insome embodiments, the active memory 3812 b may store blocks that areoutside a device's field of view. In some embodiments, the active memory3812 b may store blocks adjacent to the blocks that are in the device'sfield of view. In some embodiments, the active memory 3812 b may storeblocks that are predicted to be in the device's field of view. In someembodiments, the processor 3812 maintains a block in active memory 3812b if it is within the device's field of view at the time. The field ofview may be determined by the imaging area of one or more of thesensors. In some embodiments, the field of view may be determined by theamount of the physical world being presented to a user of the device orthat an average user may perceive, without using an AR system. Field ofview, therefore, may depend on a location of the user within thephysical world as well as the orientation of the wearable components ofthe device.

The processor 3812 may treat a block as inactive if the block becomesoutside the field of view of the device 3810 as the user moves. Inactiveblocks may be paged out of active memory to a cache. The cache may be alocal cache or a remote cache. In the embodiment of FIG. 38, blocks arefirst paged out to local cache 3814 through a local gateway 3818 b. Insome embodiments, the local cache 3814 may be the only cache available.

In some embodiments, there may be a remote cache, accessible over anetwork. In the illustrated embodiment, the cloud cache (e.g., remotecache) 3802, accessed through the network 3804 a, is an example of aremote cache. The processor 3812 may manage when blocks are movedbetween the local cache 3814 and the cloud cache 3802. For example, whenthe local cache 3814 is full, the processor 3812 may page out blocks tothe cloud cache 3802 through the network 3804 a. As blocks in the localcache 3814 are accessible for rendering a scene with lower latency thanblocks in a cloud cache, the processor 3812 may select blocks to bepaged out of the local cache 3814 using an algorithm intended to keep inthe local cache 3814 blocks most likely to become active. Such analgorithm may be based on time of access. In some embodiments, thealgorithm may be based on a prediction of motion of the device that willchange the field of view of the device.

Applications (e.g., computer games) that render scenes may obtaininformation representing the portions of the physical world that impactthe scene to be rendered. An application 3816 may obtain active blocksfrom the active memory 3812 b through a local gateway 3818 a. In someembodiments, the local gateway 3818 a may be implemented as anapplication programming interface (API), such that the processor 3812implements a “service” for the application 3816. In embodiments in whichdata of the physical world is represented as a mesh, that service may bea “meshing service.” The API may be implemented as a push or a pullinterface, or may have attributes of both. In a pull interface, forexample, the application 3816 may indicate portions of the physicalworld for which it requires data and the service may supply data ofthose portions. In a push system, for example, the service may supplydata about portions of the physical world when such data changes orbecomes available.

The portions of the physical world about which data is supplied may belimited to portions that an application 3816 has indicated are relevant,such as data within the field of view of the device or data representingportions of the physical world that is within a threshold distance ofthe field of view of the device. In a pull/push system, the application3816 may request data for a portion of the physical world and theservice may supply data on the requested portion plus any adjacentportions in which data has changed. To limit information to that whichhas changed, the service may, in addition to maintaining the blocks thatdescribed the physical world, track which versions of the blocks wereprovided to each of the applications 3816. The operations to determinewhich portion of a representation of a physical world is to be updatedand where that update occurs may be partitioned between applications3816 and a service in any suitable way. Similarly, where updated data isincorporated into a representation of the physical world can bepartitioned in any suitable ways.

In some embodiments, a reconstruction service may continuallyreconstruct a 3D representation of the physical world when the sensorsare capturing and updating data. This data may then be provided toapplications 3816 that use the 3D reconstruction data to render scenesto depict both the physical world and virtual reality objects positionedin or interacting with objects in the physical world. This data may beprovided to applications 3816 through APIs, which may be implemented asa push interface that pushes data to the application 3816 when relevantportions change, or a pull interface that responds to a request from theapplication 3816 for up-to-date information, or both.

For example, an application 3816 may operate on a mesh representation ofa portion of the physical world constituting a 45 degree viewing anglefor a distance of 10 meters relative to an origin defined by a currentlocation of a device and a direction in which the device is facing. Asthis region changes or data indicates physical changes within thisregion becomes available, a mesh may be computed to represent this area.That mesh may be computed in the application 3816 based on data suppliedby the service or may be computed in the service and provided to theapplication 3816. In either case, the service may store information inthe physical world simplifying computation of the mesh. Blocks withzippers, skirts or implemented with other techniques to facilitate“papering over” cracks between adjacent blocks, as described herein, maybe used so as to enable processing of only changed portions of therepresentation of the physical world. The changed portions of therepresentation of the physical world may then replace correspondingportions in a previous representation of the physical world.

Efficiently accessing representations of the portions of the physicalworld that are used to generate a mesh to be used by an application 3816to render a scene to a user may reduce computer resources, making XRsystems more readily implemented on portable devices or other deviceswith limited computing resources and may yield a more realistic userexperience, as the XR scene better matches the physical world.Accordingly, instead of or in addition to using blocks with skirts,zippers or other techniques to facilitated papering over cracks betweenblocks, as described elsewhere herein, an algorithm used to page blocksin and out of the active memory and/or the local cache may be selectedto reduce access time to blocks needed to compute a mesh at any giventime.

In the exemplary embodiment of FIG. 38, gateway 3818 a is a pullinterface. When an AR application 3816 requests information about aregion of the physical world, but blocks representing that region arenot in the active memory 3812 b, the processor 3812 may search blockspersisted in the local cache 3814. If the processor 3812 cannot find ablock in both the active memory 3812 b and the local cache 3814, theprocessor 3812 may search blocks persisted in the cloud cache 3802. Asaccess to the active memory 3812 b is lower latency than access to datain the local cache 3814, which is lower latency than access to data inthe cloud cache 3802, overall speed in generating a mesh may beincreased by the service implementing a paging algorithms that loadsblocks into active memory before they are requested or that moves themfrom the cloud cache 3802 to local cache 3814 before they are requested.

Similar to the first device 3810, the second device 3820 may include aprocessor 3822 having a computing portion 3822 a and an active memory3822 b, a local cache 3824, and one or more AR applications 3826. The ARapplications 3826 may communicate with processor 3822 through a localgateway 3828 a. Local cache 3824 may communication with processor 3822through a local gateway 3828 b.

Accordingly, cloud cache 3802 may persist blocks transmitted from bothdevices 3810 and 3820. The first device 3810 may access blocks capturedand transmitted from the second device 3820 in the cloud cache 3802;likewise, the second device 3820 may access blocks captured andtransmitted from the first device 3810 in the cloud cache 3802.

The devices 3801 and 3802 are provided as examples of portable ARdevices. Any suitable devices, for example, smartphones, may be used andperform similarly.

FIG. 39 is a flow chart, illustrating a method 3900 of operating asystem (e.g., the system 3700), according to some embodiments. At act3902, a device may capture 3D information about a physical worldincluding objects in the physical world and represent the physical worldas blocks including 3D reconstruction data. In some embodiments, the 3Dreconstruction data may be captured by a single system and used torender information on only that system. In some embodiments, the 3Dreconstruction data may be captured by multiple systems and may be usedto rendering information on any one of the multiple systems or on anyother systems. In these embodiments, the 3D reconstruction data from themultiple systems may be combined and made accessible to the multiplesystems or any other systems.

For example, several users, each wearing an AR system, may set theirdevices into the augmented mode while exploring a warehouse. The sensorof each device may be capturing 3D information (e.g., 3D reconstructiondata including depth maps, images, and the like) of the warehouse thatare in the sensor's file of view including objects in the warehouse(e.g., tables, windows, doors, floors, ceilings, walls). Each device maysegment the warehouse into regions having corresponding volumes, andrepresent individual regions as blocks. The blocks may have versions.Each version of the blocks may have values representing objects in aregion of the physical world at a point in time.

When an application requires information about the physical world,versions of blocks to represent that portion of the physical world maybe selected and used to generate that information. Though such selectionprocessing may be performed by any suitable processor or distributedacross any suitable processors, in accordance with some embodiments thatprocessing may be done locally at the device on which the applicationrequesting the data is executing.

Accordingly, at act 3904, a processor (e.g., processor 3812 or 3822) mayrespond to a request from an application (e.g., AR application 3816 or3826) for 3D reconstruction data. In some embodiments, regardless ofwhether an application requests 3D reconstruction data, the device(s)may continue capturing 3D information including 3D reconstruction dataabout the physical world and represent the physical world as blocks of3D reconstruction data. That 3D reconstruction data may be used tocreate new versions of the blocks.

If an application requests 3D reconstruction data, processing mayproceed to act 3906, where the processor may identify a subset of blockscorresponding to a portion of the physical world required to deliver 3Dreconstruction data in accordance with the request. The identificationof blocks may be based on, for example, data collected by a sensor(e.g., depth sensor 51, world camera 52, inertial measurement units 57,global positioning system, and/or the like). A multi-device system maycreate a common coordinate frame such that blocks generated by differentdevices associated with corresponding portions of the physical world maybe created using the common coordinate frame without regard to whichdevice provided the 3D reconstruction data to reconstruct the portion ofthe physical world represented by that block. As one example of how acommon coordinate frame may be created, data from devices in the samegeneral vicinity may be routed to the same server or one or more serversfor processing. There, data from each device may be initiallyrepresented in a device-specific coordinate frame. Once sufficient datafrom each of the devices has been gathered to identify features in acommon portion of the physical world, those features may be correlated,providing the transformation from one device-specific coordinate frameto the others. One of these device-specific coordinate frames may bedesignated as the common coordinate frame and the transformationsbetween the other coordinate frames, and that coordinate frame may beused to convert data from the device-specific coordinate frames to thecoordinate frame designated as the common coordinate frame. Regardlessof the specific mechanism by which a common coordinate frame is created,once it is created, 3D reconstruction data requested by the applicationsgenerating data for a first device may be based on updates from otherdevices, if those updates were made after any version made by the firstdevice. Blocks from both the first device and other devices may bemerged by using, for example, the common coordinate frame.

The specific processing in act 3906 may depend on the nature of therequest. In some embodiments, if the application requesting the 3Dreconstruction data maintains its own information about blocks andrequests specific blocks, the request for 3D reconstruction data at act3904 may include reference to specific subsets of blocks and identifyingthe subset of blocks at act 3906 may include determining a subset ofblocks that correspond to the specific subset of blocks. In someembodiments, the request for 3D reconstruction data at act 3904 mayinclude a reference to a field of view of the device the application isexecuting on and identifying the subset of blocks at act 3906 mayinclude determining a subset of blocks that correspond to the referencedfield of view of the device.

Regardless of the manner in which blocks are identified/determined, atact 3908, the processor may select versions of blocks of the subset ofblocks. This selection may be based on one or more criteria. Thecriteria may be based, for example, on the most recent version of theblock from the available sources. In the embodiment illustrated,versions of the block may be stored in active memory, a local cache or aremote cache. Act 3908, for example, may include selecting the versionin active memory, if it is available, or, if not, selecting the versionin local cache, if it is available, or selecting the version from theremote cache, if it is available. If no version of the block isavailable, then selection may entail generating the block, such as forexample, from the data (e.g., 3D reconstruction data) collected with asensor (e.g., depth sensor 51, world camera 52, and/or inertialmeasurement units 57). Such an algorithm for block selection may be usedin a system that manages, such as a background process, block versionsstored in each of the possible locations. An exemplary managementprocess is described below in connection with FIG. 41.

At act 3910, the processor may provide information based on the selectedversions of blocks to the application. Processing at act 3910 may entailsimply providing the blocks to the application, which may be appropriatewhen the application uses blocks directly. Where the applicationreceives a mesh, processing at act 3910 may entail generating a meshfrom the blocks and/or the subset of blocks, and providing the mesh, orany suitable portion of the mesh, to the application.

FIG. 40 is a flow chart, illustrating details capturing 3D informationabout objects in a physical world and representing the physical world asblocks of 3D reconstruction data, according to some embodiments. In someembodiments, FIG. 40 is a flow chart illustrating details of act 3902 ofFIG. 39. At act 4002, one or more sensors (e.g., depth sensor 51, worldcamera 52, inertial measurement units 57, and/or the like) of a system(e.g., system 3700) capture 3D information about objects in a physicalworld including objects in the physical world.

At act 4004, a processor (e.g., processor 3812 or 3822) of the systemmay create versions of blocks including 3D reconstruction data of thephysical world based on the 3D information captured by the one or moresensors. In some embodiments, each block may be formatted as one or moreportions of a mesh. In some embodiments, other representations of thephysical world may be used.

The blocks may have versions, such that each time information about aregion of the physical world is captured by any device, a new version ofthe block may be stored. Each version of the block may have 3Dreconstruction data including values representing objects in a region ofthe physical world at a point in time. In some embodiments, suchprocessing may be performed locally on the device, resulting in newversions of blocks being stored in active memory. In some embodiments,in a multi-device system, similar processing may be performed in aserver (e.g., server 3710 of FIG. 37), which may manage versions of theblocks such that the most recent version of each block available in itsremote cache is supplied when requested by any device.

Because the blocks represent a physical world, much of which will beunchanging, a new version of a block may not necessarily be created whennew 3D reconstruction data representing the corresponding region of thephysical world is available. Rather, managing versions of the blocks mayentail processing of the 3D reconstruction data representing thephysical world to determine whether there have been sufficient changessince the last version of the blocks representing those regions of thephysical world to warrant changes. In some embodiments, sufficientchanges may be indicated by a magnitude of a block metric becoming abovea threshold value since the last version has been stored.

In some embodiments, other criteria may be applied to determine whichversion of a block is supplied as the current version when a block isrequested, such as the version that has the lowest value of a metricindicating completeness or accuracy of the data in the block. Similarprocessing may be performed on each device, resulting in versions ofblocks stored in the local cache on the devices.

One or more techniques may be used to manage the versions of a blockavailable to a service on each device. If, for example, there is anacceptable version of the block already computed, rather than created anew version of the block from the sensor data, a processor may access apreviously stored block. Such access may be performed efficiently bymanaging the storage of versions of the blocks. At act 4006, theprocessor of the device may page the versions of the blocks of 3Dreconstruction data of the physical world out of an active memory (e.g.,active memory 3812 b or 3822 b). The paging may include the processoraccessing sensor data so as to continuously update the blocks in activememory/local cache/cloud cache, for example, as a function of the fieldof view of that device. When the field of view of the device changes,blocks corresponding to the new field of view may be transferred (e.g.,paged) from the local cache and/or the cloud cache into the activememory, and blocks corresponding to areas just outside the new field ofview (e.g., blocks adjacent to the blocks in the new field of view) maybe transferred (e.g., paged) from the active memory and/or the cloudcache to the local cache. For example, at act 4008, the versions of theblocks paged out by the processor may be persisted in a local memory(e.g., local cache 3814 or 3824) and/or a remote memory (e.g., cloudcache 3802). In some embodiments, as each new version of a block iscreated, for example, on a device, that version may be sent to remotememory such that other users can access it.

FIG. 41 is a flow chart illustrating an exemplary process for performingselecting versions of blocks representing a subset of blocks, accordingto some embodiments. In some embodiments, FIG. 41 is a flow chartillustrating details of act 3908 of FIG. 39. For selecting a version ofeach block within the subset of blocks, at act 4102, a processor (e.g.,processor 3812 or 3822) may query whether an up-to-date version isstored in active memory (e.g., active memory 3812 b or 3822 b). In someembodiments, whether a version is up-to-date may be determined bycomparing a value attached to the version (e.g., a geometry changemagnitude, a timestamp, and the like) with data collected with a sensor(e.g., depth sensor 51, world camera 52, and/or inertial measurementunits 57). In some embodiments, a comparison between current sensor dataand the version of the block stored in active memory may be made. Basedon a degree of difference, which may represent a changes in the physicalworld or quality of the version in active memory, for example, theversion in active memory may be deemed up-to-date.

If an up-to-date version is stored in the active memory, processingproceeds to act 4104, where the up-to-date version is selected. If noup-to-date version is stored in the active memory, processing proceedsto act 4106, where the processor may query whether an up-to-date versionis stored in a local memory (e.g., local cache 3814 or 3824). That querymay be performed using criteria described above as in connection withact 4102, or any other suitable criteria. If an up-to-date version isstored in the local memory, at act 4108, the up-to-date version isselected.

If no up-to-date version is stored in the local memory, at act 4110, theprocessor may query whether an up-to-date version is stored in a remotememory (e.g., cloud cache 3802). That query also may be performed usingcriteria described above as in connection with act 4102, or any othersuitable criteria. If an up-to-date version is stored in the remotememory, at act 4112, the up-to-date version is selected.

If no up-to-date version is stored in the remote memory, processing mayproceed to act 4114 where the processor of the device may generate a newversion of the block based on 3D information (e.g., the 3Dreconstruction data) captured by the sensors. In some embodiments, atact 4116, the processor may identify neighboring blocks of the blockwith a new version, and update the identified neighboring blocksaccording to the new version of the block.

FIG. 42 is a flow chart, illustrating a method 4200 of operating asystem, according to some embodiments. In the method 4200, rather thanpulling blocks into active memory and/or local cache when the devicerequires those blocks, paging may be managed based on projections of thefield of view of the device based on device motion.

Similar to act 3902, at act 4202, sensors on a device may capture 3Dinformation about a physical world including objects in the physicalworld and represent the physical world as blocks including 3Dreconstruction data.

At act 4204, a processor (e.g., processor 3812 or 3822) may compute,based at least in part on an output of the sensor, a region of thephysical world at which a portable pointable component (e.g., depthsensor 51, world camera 52, and/or inertial measurement units 57) willbe pointed at a future time. In some embodiments, the processor maycompute based on motion data from inertial sensors or analysis resultsof captured images. In a simple calculation, for example, to get a fastresult, the processor may compute based on translation and rotation ofthe head of a user. In applying more comprehensive algorithms, theprocessor may compute based on objects in the scene. For example, thealgorithms may consider a user walking towards a wall or table isunlikely to walk through the wall or table.

At act 4206, the processor may select blocks based on the computedregion. At act 4208, the processor may update an active memory (e.g.,active memory 3812 b or 3822 b) with the selected blocks. In someembodiments, the processor may select blocks based on the flow chart ofFIG. 41. At act 4210, the processor may select blocks from the activememory to supply an application (e.g., application 3816 or 3826), forexample, through an API, based on changes to each of the blocks since aversion of the block was last supplied to the application.

In some embodiments, at act 4206, the processor may request the selectedblocks from a remote memory (e.g., cloud cache 3802), and updateinformation stored in a local cache (e.g., 3814 or 3824) such that thelocal cache stores the selected blocks. Act 4206 may be similar to act3908 described in FIG. 39.

Block-based processing as described above may be based on blocks thatallow portions of a 3D representation to be processed separately andthen combined with other blocks. In accordance with some embodiments,the blocks may be formatted such that, when a block is changed, thechanged representation largely or wholly maintains the values of theblock at the interfaces to other blocks. Such processing enables thechanged version of a block to be used with versions of adjacent blocksthat were not changed, without creating unacceptable artifacts in ascene rendered based on changed and unchanged blocks. FIGS. 43A-48illustrate such blocks.

A 3D representation of a physical world may be provided by a volumetric3D reconstruction, which may create a 3D reconstruction data hierarchyof 3D information of the physical world captured by a sensor. Forexample, the sensor may be a depth camera, which may capture 3Dinformation of the physical world, for example, a stream of depth imageswith respective poses of the depth camera (i.e. camera poses). The 3Dinformation of the physical world may be processed into a voxel grid.Each voxel may contain one or more signed distance functions (SDFs) thatdescribe whether the voxel lies inside or outside the geometries ofobjects in the physical world. The voxels may be grouped into “bricks.”Each brick may include multiple voxels, for example, in cubic volumessuch as 8³ voxels. The bricks may be further grouped into “tiles.” Eachtile may include multiple bricks.

In some embodiments, the voxel grid may be mapped to conform to a memorystructure. A tile may correspond to a memory page of a storage medium.The size of a tile may be variable, for example, depending on the sizeof a memory page of a storage medium in use. Accordingly, the 3Dreconstruction data may be transmitted among storage mediums (e.g., anactive memory and/or local memory of a device, and/or a remote memory ina cloud), on the basis of tiles. In some embodiments, one or more tilesmay be processed to generate a block. A block may be updated, forexample, when at least one voxel in the one or more tiles changes.

A block may not necessarily be limited to corresponding to tiles. Insome embodiments, a block may be generated from one or more bricks, oneor more voxels, or one or more SDF samples, and the like. A block may beany suitable partition of a physical world. A block may not necessarilylimited to be in the format of a mesh. A block may be in any suitableformat of 3D reconstruction data.

FIG. 43A-D illustrate an exemplary physical world 4300 represented bymesh blocks 4302. Each mesh block 4302 may be extracted from voxels 4304corresponding to a predetermined volume of a mesh block. In theillustrated example, each block may be the output of a cubed region(e.g., 1 m³) of voxels in the low level reconstruction model. Each meshblock 4302 may contain a part of the world mesh and may be treatedindependently. This may enable scalability with fast local updates as aresult of some blocks changed when exploring new areas or things movedin the environment. In the illustrated example, there are no changes tothe mesh blocks except for a mesh block 4306, which has a new object4308 placed in front of the existing surface 4310. In this case, the ARsystem needs update only the mesh block 4306, which could savesignificant computing power compared to arbitrarily update the wholemesh for a world.

FIG. 43B is a simplified schematic diagram illustrating a mesh block,according to some embodiments. In the illustrated example, a mesh blockinternally may have a fully connected mesh, which means that verticesare shared by multiple triangles.

On the other hand, individual mesh blocks may be independent meshes,which are not connected. FIG. 43C is a simplified schematic diagramillustrating a crack, which may exist at edges of two adjacent meshblocks, according to some embodiments. FIG. 44D is a simplifiedschematic diagram illustrating the crack in FIG. 43C being papered overby implementing mesh skirts that overlap an adjacent mesh blocks,according to some embodiments.

FIG. 44 is a schematic diagram, illustrating in 2D segmenting arepresentation 4400 of a portion of a physical world, according to someembodiments. The 2D representation 4400 may be obtained by connecting aset of four blocks: Blocks A-D. The representation 4400 may be segmentedinto four blocks: Blocks 4402, 4404, 4406, and 4408, for example,illustrated by solid lines 4410. In some embodiments, Blocks 4402, 4404,4406, and 4408 may be designated as Blocks A-D respectively.Applications may desire 3D reconstruction data in the format of meshesfor further processing such as occlusion testing, and generatingphysical effects in physical engines. In some embodiments, the set ofblocks may be in the format of meshes, which may be generated by adevice (e.g., the devices 3810, 3820), a network (e.g., a cloud thatincludes cloud cache 3802), or a discrete application (e.g., theapplications 3816, 3826).

In some embodiments, regions at the boundaries of Blocks 4402, 4404,4406, and 4408 may be skirts, for example, illustrated by dashed lines4412. In some embodiments, each of Blocks A-D may include a block and acorresponding skirt. For example, Block B may include Block 4404 and askirt 4414 that overlaps with boundary parts of Block B's neighboringblocks A, C, and D such that cracks between the blocks may be paperedover when the blocks are connected into a global mesh. Blocks A, C, andD may also include corresponding skirts. Accordingly before returningblocks including 3D reconstruction data to an application, a processormay paper over any cracks between the blocks.

In some embodiments, the global mesh may be a topologically-connectedglobal mesh. For example, adjacent blocks in the set of blocks may sharemesh vertices at block boundaries such as lines 4410. In someembodiments, the global mesh may be visually seamless althoughtopologically-disconnected using any suitable techniques such as skirtsand zippers.

Although a method employing skirts is illustrated, other method may beused to enable a changed block to be combined with an unchanged adjacentblock, such as zippers. Although in the illustrated example, a portionof a physical world is represented by four 2D blocks, it should beappreciated that a portion of a physical world may be represented by anysuitable number of 2D and/or 3D blocks, for example, two, three, five,six or more. Each block may correspond to a space in the physical world.In some embodiments, blocks in a 2D and/or 3D representation of aportion of a physical world may correspond to spaces of the same size(e.g., area/volume) in the physical world. In some embodiments, blocksin a 2D and/or 3D representation of a portion of a physical world maycorrespond to spaces of different sizes in the physical world.

FIG. 45 is a schematic diagram, illustrating a 3D representation 4500 ofa portion of a physical world, according to some embodiments. Similar tothe 2D representation 4400, 3D representation 4500 may be obtained byconnecting eight blocks: Blocks A-H. In some embodiments, Blocks A-H maybe exclusive to each other, for example, having no overlapped region. Insome embodiments, Blocks A-H may have regions overlapping a neighboringblock (e.g., skirts 4516). In some embodiments, each of Blocks A-H mayhave versions. Each version of a block may have values representingobjects in a region of the physical world at a point in time. In theillustrated example, 3D representation 4500 includes versions of BlocksA-H: version 4502 of Block A, version 4504 of Block B, version 4514 ofBlock C, version 4512 of Block D, version 4534 of Block E, version 4506of Block F, version 4508 of Block G, and version 4510 of Block H.Version 4502 of Block A may include value 4518; version 4504 of Block Bmay include value 4522; version 4514 of Block C may include value 4528;version 4512 of Block D may include value 4532; version 4534 of Block Emay include value 4520; version 4506 may include value 4524; version4508 may include value 4526; and version 4510 may include value 4530.

FIG. 46 is a schematic diagram, illustrating a 3D representation 4600 ofa portion of a physical world obtained by updating the 3D representation4500, according to some embodiments. Compared to 3D representation 4500,3D representation 4600 may have a new version 4610 of Block H thatincludes information 4630. Information 4630 may be different frominformation 4530. For example, a first device may persist version 4510of Block H in a remote memory. Version 4510 of Block H may includeinformation 4530 corresponding to a table with an empty surface. Afterthe first device leaves the area (e.g., the field of view of the firstdevice no longer includes Block H), a second device may place a virtualand/or physical box on the table's surface and then persist version 4610of Block H in the remote memory. Version 4610 of Block H may includeinformation 4630 corresponding to the table with the virtual and/orphysical box. If the first device returns, the first device may becapable of selecting version 4610 of Block H to view from availableversions of Block H including version 4610 and version 4510 of Block H.

FIG. 47 is a schematic diagram, illustrating an augmented world 4700viewable by a first device 4702 (e.g., device 3810) and a second device4712 (e.g., device 3820). First and second devices may include ARdisplay systems 4704 and 4714 (e.g., AR display system 80) operating inan augmented mode. An augmented world 4700 may be obtained by connectingfour blocks: Blocks A-D. In the illustrated example, the augmented world4700 includes versions of Blocks A-D: version 4702A of Block A, version4702B of Block B, version 4702C of Block C, and version 4702D of BlockD. The first device 4702 may be looking at a first direction 4706 andhave a first field of view (FOV) 4708. In the illustrated example, thefirst FOV includes version 4702B of Block B and version 4702D of BlockD. A processor (e.g., 3812) of the second device 4704 may includecomputer-executable instructions for identifying Blocks B and Dcorresponding to the first FOV and selecting version 4702B of Block Band version 4702D of Block D. The second device 4714 may be looking at asecond direction 4716 and have a second FOV 4718. In the illustratedexample, the second FOV includes version 4702C of Block C and version4702D of Block D. A processor (e.g., 3822) of the second device 4714 mayinclude computer-executable instructions for identifying Blocks C and Dcorresponding to the second FOV and selecting version 4702C of Block Cand version 4702D of Block D.

FIG. 48 is a schematic diagram, illustrating an augmented world 4800obtained by updating augmented world 4700 with new versions of blocks,according to some embodiments. Compared to the augmented world 4700, theaugmented world 4800 may include version 4802C of Block C different fromversion 4702C, and version 4802D of Block D different from version4702D. The first device 4702 may be looking at a third direction 4806and have a third FOV 4808. In the illustrated example, the third FOVincludes version 4802C of Block C and version 4802D of Block D. Theprocessor of the first device 4702 may include computer-executableinstructions for determining which of versions 4702C, 4802C, 4702D, and4802D to provide to an application based on, for example, the change ofFOV and/or information collected with a sensor of the first device 4702.In some embodiments, the processor of the first device 4702 may includecomputer executable instructions for generating version 4802C of Block Cand version 4802D of Block D when there are no corresponding up-to-dateversions of Blocks C and D in a local memory (e.g., local cache 3814,3824) or a remote memory (e.g., cloud cache 3802). In some embodiments,the first device 4702 may be capable of estimating changes in its FOV(e.g., from the first FOV 4708 to the third FOV 4808, selecting Block Cbased on the estimation, and storing versions of Block C to a memorycloser to the processor (e.g., moving versions of Block C from a remotememory to a local cache, or from a local cache to an active memory).

Method of Occlusion Rendering Using Raycast and Live Depth

The realism with which AR and MR scenes are presented to a user may beenhanced by providing occlusion data to applications generating suchscenes, where the occlusion data is derived from a combination of one ormore depth data sources. The occlusion data may represent surfaces ofphysical objects in the scene and may be formatted in any suitable way,such as by depth data indicating a distance from a point of view fromwhich the scene is to be rendered and a surface. For example, usecomponents 164 may receive occlusion data from perception module 160(FIG. 6).

However, in some embodiments, one data source may be one or more depthcameras that directly sense and capture position between the depthcameras and real objects in a physical world. Data from depth camerasmay be provided directly to use components 164 or may be providedindirectly, such as through perception module 160. The one or more depthcameras may provide an immediate view of the physical world at framerate that is sufficiently high to capture changes in the physical world,but low enough not to burden processing. In some embodiments, the framerate may be 5 frames per second, 10 frames per second, 12 frames persecond, 15 frames per second, 20 frames per second, 24 frames persecond, 30 frames per second, and the like. In some embodiments, theframe rate may be less than 5 frames per second. In some embodiments,the frame rate may be more than 30 frames per second. Accordingly, insome embodiments, the frame rate may be in the range of 1-5 frames persecond, 5-10 frames per second, 10-15 frames per second, 15-20 framesper second, or 20-30 frames per second, and the like.

A second source of data may be stereoscopic visual cameras that maycapture visual representations of the physical world. Depth data fromthe depth camera and/or image data from the visual camera may beprocessed to extract points representing the real objects in thephysical world. Images from the visual camera, such as a stereoscopiccamera, may be processed to compute a three-dimensional (3D)reconstruction of the physical world. In some embodiments, depth datamay be generated from the images from the visual cameras, for example,using deep learning techniques. Some or all of the 3D reconstruction maybe computed before occlusion data and may be stored in memory. In someembodiments, the 3D reconstruction may be maintained in computer memoryby a process independent of any process that generates depth informationfor occlusion processing, which may access that stored 3D reconstructionas needed. In some embodiments, the 3D reconstruction may be maintainedin memory, and portions of it may be updated in response to anindication, for example, computed based on depth information, that therewas a change in the physical world corresponding to the portion of the3D reconstruction. In some embodiments, a second data source may beimplemented by a raycast into the 3D reconstruction of the physicalworld to obtain low-level 3D reconstruction data (e.g., a raycast pointcloud). With a raycast, the data from the second data source may beselected to fill any holes in the occlusion data, enabling the data fromthe two (or more) sources to be integrated.

In accordance with some embodiments, the depth data and/or the imagedata and/or the low-level data of the 3D reconstruction may be orientedwith respect to a user of an AR or MR system. Such an orientation may beachieved, for example, by using data from sensors worn by the user. Thesensors may be worn, for example, on a head-mounted display device/unit.

In a system in which the occlusion data may be generated from more thanone depth data source, the system may include a filter that identifieswhich portions of a 3D region are represented by data from each ofmultiple depth data sources. The filter may apply one or more criteriato identify portions of the region, for which data from a second datasource is to be collected. These criteria may be indications of thereliability of the depth data. Another criteria may be changes detectedin the portion of the region since depth data was collected.

Selecting between multiple depth data sources to provide data fordifferent portions of a representation of a 3D region may reduceprocessing time. For example, when less processing is required to deriveocclusion data from data collected with the first data source than witha second data source, the selection may favor data from the first datasource, but use data from the second source when data from the firstsource is not available or acceptable. As a specific example, the firstdata source may be a depth camera and the second data source may be astereoscopic visual camera. The data from the stereoscopic camera may beformatted as a 3D reconstruction of the physical world. In someembodiments, the 3D reconstruction may be computed before occlusion datais required. Alternatively, or additionally, the 3D reconstruction maybe re-computed when occlusion data is required. In some embodiments,criteria may be applied to determine whether the 3D reconstructionshould be recomputed.

In some embodiments, this occlusion data is computed by a service,providing occlusion data through an application programming interface(API) to applications executing on a computing device that will renderXR scenes. The service may execute on the same computing device as theapplications or may execute on a remote computer. The service mayinclude one or more of the components discussed herein, such as thefilter for data from a first data source, and/or an engine toselectively fetch data from a second data source based on the filtereddata from the first data source. The service may also include acomponent to combine the filtered data from the first data source andthe selected data from the second data source to generate occlusiondata.

The occlusion data may be formatted in any suitable way that representssurfaces in the physical world. For example, the occlusion data may beformatted as depth buffers of surfaces, storing data identifyinglocations of the surfaces in the physical world. This occlusion data maythen be used in any suitable way. In some embodiments, the occlusiondata may be provided to one or more applications wanting to have virtualobjects occluded by real objects. In some embodiments, the occlusiondata may be formatted as a depth filter created by the system for anapplication requesting an occlusion service for occlusion data forrendering virtual objects at one or more locations. The depth filter mayidentify locations for which the application should not render imageinformation for the virtual object because virtual objects in thoselocations would be occluded by surfaces in the physical world. It shouldbe appreciated that “occlusion data” may be in a suitable format toprovide information about surfaces in the physical world and need not beused for occlusion processing. In some embodiments, occlusion data maybe used in any application performing processing based on arepresentation of surfaces in a scene of the physical world.

Approaches as described herein provide occlusion data with less latencyand/or using lower computing resources than in a conventional AR and MRsystem in which an application uses mesh data to perform occlusionprocessing. The mesh data may be obtained by processing geometrical dataextracted by an image sensor with multiple time or cost intensive stepsincluding, marching cube algorithms, mesh simplification, and applyingtriangle count limits. The mesh data may take from hundreds ofmilliseconds to seconds to calculate, and a delay in having anup-to-date mesh may lead to visible artifacts when the environment isdynamically changing and an application renders a scene using anoutdated mesh. These artifacts manifest, for example, as virtual contentappearing superimposed on top of real objects when the virtual contentis supposed to be rendered behind the real object, which breaks theperception/feeling of immersion for the user of such an application andprovides the user with incorrect cues for 3D depth perception.

For an application using a mesh for occlusion processing to have anup-to-date mesh, the application has to either continuously query formeshes (causing significant continuous processing to occur) or make useof a mechanism to determine if there have been changes and then queryfor a new mesh (which would reduce the overall processing, but stillhave high latency between changes in the physical world and when a meshreflecting those changes arrived at the application).

By making use of low-level data of 3D reconstruction data (e.g., pointclouds) and live depth data directly for occlusion instead of a mesh, itis possible to decrease the latency between change occurring in theenvironment and being reflected in occlusion data, maintaining a closersynchronization with the physical world and hence higher perceivedvisual quality.

In some embodiments, real-time depth maps of a physical environment maybe obtained from depth sensors (e.g., depth cameras). Each pixel in adepth map may correspond to a discrete distance measurement capturedfrom a 3D point in the environment. In some embodiments, these depthcameras may provide depth maps including a set of points at real-timerates. However, depth maps may have holes, which may result from thedepth camera being unable to acquire sensor data representing a regionor acquiring incorrect or unreliable data representing a region. In someembodiments, if the depth sensor uses infrared (IR) light, the holes mayresult, for example, from materials or structures in the physicalenvironment that do not reflect IR light very well or at all. In someembodiments, the holes may result, for example, from very thinstructures or surfaces at glancing incidence angles that do not reflectlight toward the depth sensor. The depth sensor may also experiencemotion blur when moving fast, which may also lead to missing data.Further, “holes” in the depth map represent regions of the depth mapthat for any other reason are not suitable for use in occlusionprocessing. Such holes may be detected using any suitable processing,such as processing the depth map to detect lack of connectivity betweenpoints or regions in the depth map. As another example, holes may bedetected using processing that computes quality metrics for regions ofthe depth map and processing regions with low quality metrics as holes.One such metric may be image to image variation for pixels in a depthmap representing the same location in the physical world. Pixels withsuch a variation exceeding a threshold may be classified as a hole. Insome embodiments, holes may be identified by pixels meeting predefinedstatistical criteria for a cluster of pixels in which the qualitymetric(s) is below a threshold.

In some embodiments, a depth map may first be “filtered” to identifyholes. Then rays from the point of view from which the scene will berendered to the holes may be determined. Those rays may be “cast” into a3D representation of the physical world, created using sensors otherthan the depth sensor alone, to identify data representing the regionsof the holes. The 3D representation of the physical world, for example,may be a 3D reconstruction created from data from a stereoscopic visualcamera. The data from the 3D reconstruction identified by such raycasting may be added to the depth map, thereby filling the holes.

The 3D reconstruction may be computed from image sensor data when holesare identified. Alternatively, some or all of the 3D reconstruction maybe computed in advance and stored in memory. The 3D reconstruction, forexample, may be maintained in computer memory by a process independentof any process that generates depth information for occlusionprocessing, which may access that stored 3D reconstruction as needed. Asa further alternative, the 3D reconstruction may be maintained inmemory, but portions of it may be updated in response to an indication,computed based on depth information, that there was a change in thephysical world corresponding to the portion of the 3D reconstruction.

In an XR system, the rays may have a same pose an eye gaze of a user. Inexemplary systems as described below, the depth map may similarly beacquired with the same eye gaze of the user, because the depth sensormay be worn by the user, and it may be mounted on the user's head nearthe eyes. A visual camera used to form the 3D reconstruction data maysimilarly be worn by a user such that the images, and data derived fromthose images, may be related to a coordinate system that enables rays,defined with respect to the depth map, to be cast into the 3Dreconstruction computed from the visual images. An inertial measurementunit and/or other sensors, similarly worn by the user and/or associatedwith the sensors, may provide data to perform coordinate transformationsto add data to the 3D representation regardless of the pose of thevisual camera and to relate rays, defined with respect to the depth map,to the 3D reconstruction.

In some embodiments, a user's focus or related virtual content placementinformation may guide raycast to make it adaptive in the image space bycasting more dense rays at depth discontinuities to obtain high qualityocclusion at the object boundaries, and sparse rays in the center ofobjects in view to decrease processing requirements. Raycast mayadditionally give local 3D surface information such as normal andposition, which may be used to enable improved timewarp processing withthe depth information and to alleviate missing visible pixels that needto be rendered or raytraced in typical rendering engines. Timewarp is atechnique in XR that modifies a rendered image before sending it to adisplay to correct for a head movement that is computed to occur betweenrendering and displaying. In some embodiments, timewarping may be usedto synchronize the data from the depth map and the 3D representation ofthe physical world that may be used to generate data to fill holes inthe depth map. Data from both data sources may be timewarped torepresent the computed pose at the time of displaying. In someembodiments, the data from the 3D representation may be timewarped torepresent the computed pose at the time data is captured with the depthmap.

In some embodiments, advanced features such as timewarp may utilize the3D local surface information from raycast. When the content frame isrendered without physical world occlusion or with eroded depth imaging,timewarp may fill all the missing visible pixels that were previouslyoccluded. Therefore, there may be no need for a rendering engine to fillthe pixels, enabling more loosely decoupled rendering applications (ormore independent timewarp).

Processing as described above may be performed in hardware processors ofmany suitable forms on data acquired with many suitable sensors andpresented on many suitable interfaces. Examples of suitable systems,including sensors, processing and user interfaces, are presented below.In the illustrated embodiment, a “service” may be implemented as part ofan XR system with computer executable instructions. Execution of thoseinstructions may control one or more processors to access to sensor dataand then generate and supply depth information to applications executingon the XR system. Those instructions may be executed on the sameprocessor or same device that executes the application presenting XRscenes to a user or may be on a remote device, accessed by the userdevice over a computer network.

FIG. 49 illustrates an occlusion rendering system 4900, according tosome embodiments. The occlusion rendering system 4900 may include areconstruction filter 4902. The reconstruction filter 4902 may receivedepth information 4904. In some embodiments, the depth information 4904may be a sequence of depth images captured by a depth camera. In someembodiments, the depth information 4904 may be derived from a sequenceof images captured by visual cameras, for example, using structure frommotion based on a single camera and/or using stereo computation based ontwo cameras. FIG. 50 illustrates a depth image 5000, according to someembodiments. In some embodiments, surface information may be generatedfrom the depth information. The surface information may indicatedistance to the physical object in a field of view (FOV) of ahead-mounted display device comprising a depth camera and/or visualcameras. The surface information may be updated in real time as a sceneand FOV changes.

A second source of depth information is illustrated as 3D reconstruction4908. The 3D reconstruction 4908 may include a 3D representation of thephysical world. The 3D representation of the physical world may becreated and/or maintained in computer memory. In some embodiments, the3D representation may be generated from images captured by visualcameras, for example, using structure from motion based on a singlecamera and/or using stereo computation based on two cameras. In someembodiments, the 3D representation may be generated from depth imagescaptured by a depth camera. For example, the 3D reconstruction 4908 maybe created and/or updated using the depth information 4904 incombination with a pose of the depth camera with respect to a worldorigin. That representation may be built up, and modified over time, forexample as a user, wearing the camera looks around the physical world.In some embodiments, the depth information 4904 also may be used togenerate a 3D representation of a physical world. The 3D reconstruction4908 may be a volumetric reconstruction that includes 3D voxels. In someembodiments, each 3D voxel may represent a cube of space (e.g., 0.5meters by 0.5 meters by 0.5 meters) and each 3D voxel may include datarelated to and/or describing surfaces in the real-world in that cube ofspace.

The 3D reconstruction 4908 of the world may be stored in any suitableway. In some embodiments, the 3D reconstruction 4908 may be stored as a“cloud” of points representing features of objects in the physicalworld. In some embodiments, the 3D reconstruction 408 may be stored as amesh, with groups of points defining vertices of triangles thatrepresent surfaces. In some embodiments, the 3D reconstruction 4908 maybe generated using other techniques such as room layout detectionsystem, and/or object detection. In some embodiments, a number oftechniques may be used together to generate the 3D reconstruction 4908.For example, object detection may be used for known physical objects inthe physical world, 3D modeling may be used for unknown physical objectsin the physical world, and room layout detection system may also be usedto identify the boundaries in the physical world such as walls andfloors.

The reconstruction filter 4902 may include computer executableinstructions for producing a depth map based on the depth information4904. The depth map may include one or more pixels. Each pixel mayindicate a distance to a point of a surface in a physical world. In someembodiments, the reconstruction filter 4902 may composite depthinformation 4904 and data from the raycast engine 4906. In someembodiments, the reconstruction filter 4902 may reduce or remove noisefrom the depth information 4904 based, at least in part, on the datafrom the raycast engine 4902 and/or from the composite of the depthinformation 4904 and the data from the raycast engine 4906. In someembodiments, the reconstruction filter 4902 may up-sample the depthinformation 4904 using deep-learning techniques.

The reconstruction filter 4902 may identify regions of the depth mapbased on a quality metric. For example, when the quality metric of apixel is above a threshold, the pixel may be determined as incorrect ornoisy. Regions of the depth map containing the incorrect or noisy pixelsmay be referred to as holes (e.g., holes 5002).

FIGS. 54A and 54B provide an alternative example of how holes mightresult in a depth map, in embodiments in which the depth map isconstructed from multiple depth images. FIG. 54A is a sketch of a regionbeing imaged with a depth camera from a first point of view to identifyvoxels that are occupied by a surface and voxels that are empty. FIG.54B is a sketch of a region being imaged with a depth camera frommultiple points of view to identify voxels that are occupied by asurface and are empty. FIG. 54B illustrates multiple voxels determinedto be occupied by a surface or empty by fusing data from multiple cameraimages. However, voxels in region 5420 have not been imaged. Region 5420could have been imaged with the depth camera in position 5422, but thecamera has not moved to that location. Accordingly, region 5420 is anobserved space, for which no volumetric information is available. The ARsystem may guide the user wearing it to scan the observed space.

Referring back to FIG. 49, the reconstruction filter 4902 may inform araycast engine 4906 about locations of the holes. The raycast engine4906 may generate a view of the physical world given a user's pose, andmay remove the holes out of the depth map. The data may representportions of a user's current view of the physical world at a currenttime, or, where time warping is used, at a time to which the data is tobe time warped. The raycast engine 4906 may generate one or more 2Dimages, for example, one image per eye. In some embodiments, thereconstruction filter 4902 may remove regions of the depth map that arespaced from a location of a virtual object beyond a threshold distancebecause these regions may not be relevant to an occlusion test for thevirtual object.

The raycast engine 4906, which generates the view of the physical worldgiven the user's pose, may be implemented by any suitable techniques. Insome embodiments, the raycast engine 4906 may implement ray-castingalgorithm on the 3D reconstruction 4908 to extract data from it. Theray-casting algorithm may take the user's pose as input. The raycastengine 4906 may cast rays from a virtual camera to the 3D reconstruction4908 of the physical world to obtain surface information that is missingfrom the depth map (e.g., holes). The raycast engine 4906 may cast denserays at boundaries of physical objects in the physical world to obtainhigh quality occlusion at the object boundaries, and sparse rays incenter regions of the physical objects to decrease processingrequirements. The raycast engine 4906 may then provide the raycast pointclouds to the reconstruction filter 4902. The raycast engine 4906 isillustrated as an example. In some embodiments, the raycast engine 4906may be a meshing engine. The meshing engine may implement a meshingalgorithm on the 3D reconstruction 4908 to extract data from it, forexample, including triangles and connectivity of the triangles. Themeshing algorithm may take the user's pose as input.

The reconstruction filter 4902 may composite the depth information 4904and data from the raycast engine 4906, compensating for the holes in thedepth map from the depth information 4904 with the data from the raycastpoint clouds from the raycast engine 4906. In some embodiments, theresolution of the depth map may be improved. This method may be used togenerate a high resolution depth image from sparse or low resolutiondepth image.

The reconstruction filter 4902 may provide the updated depth map to anocclusion service 4910. The occlusion service 4910 may compute occlusiondata based on the updated depth map and information about a location ofa virtual object in the scene. The occlusion data may be depth buffersof surfaces in the physical world. The depth buffers may store depths ofpixels. In some embodiments, the occlusion service 4910 may be aninterface with applications 4912. In some embodiments, the occlusionservice 4910 may interface with a graphics system. In these embodiments,the graphics system may expose a depth buffer to the applications 4912,where the depth buffer is pre-filled with the occlusion data.

The occlusion service 4910 may provide the occlusion data to one or moreapplications 4912. In some embodiments, the occlusion data maycorrespond to the user's pose. In some embodiments, the occlusion datamay be a pixel-wise representation. In some embodiments, the occlusiondata may be a mesh representation. The applications 4912 may beconfigured to execute computer executable instructions to render virtualobjects in scenes based on the occlusion data. In some embodiments,occlusion rendering may be conducted by a separate graphic systeminstead of the applications 4912. The separate graphic system may usetimewarp techniques.

In some embodiments, the reconstruction filter 4902, raycast engine4906, and occlusion service 4910 may be a remote service, for example,remote processing module 72; the 3D reconstruction 4908 may be stored ina remote memory, for example, remote data repository 74; and theapplications 4912 may be on an AR display system 80.

FIG. 51 illustrates a method 5100 of occlusion rendering in an augmentedreality (AR) environment, according to some embodiments. At act 5102,depth information may be captured from a depth sensor (e.g., depthsensor 51), for example, on a head-mounted display device. The depthinformation may indicate distance between the head-mounted displaydevice and a physical object. At act 5104, surface information may begenerated from the depth information. The surface information mayindicate distance to the physical object in a field of view (FOV) of thehead-mounted display device and/or a user of the head-mounted displaydevice. The surface information may be updated in real time as a sceneand FOV changes. At act 5106, portions of a virtual object to render maybe computed from the surface information and information about alocation of the virtual object in the scene.

FIG. 52 illustrates details of act 5104, according to some embodiments.At act 5202, the depth information may be filtered to generate a depthmap. The depth map may include one or more pixels. Each pixel mayindicate a distance to a point of the physical object. At act 5204,low-level data of a 3D reconstruction of the physical object may beselectively acquired from, for example, 3D reconstruction 4908. At act5206, the surface information may be generated based on the depth mapand the selectively-acquired low-level data of the 3D reconstruction ofthe physical object.

FIG. 53 illustrates details of act 5202, according to some embodiments.At act 5302, quality metrics for regions of the depth map may bedetermined. The quality metrics may indicate whether regions of thedepth map are incorrect or noisy. At act 5304, holes in the depth mapmay be identified based on the quality metrics by, for example,comparing to a threshold value. At act 5306, the identified holes may beremoved out of the depth map.

CONCLUSION

Having thus described several aspects of some embodiments, it is to beappreciated that various alterations, modifications, and improvementswill readily occur to those skilled in the art.

As one example, embodiments are described in connection with anaugmented (AR) environment. It should be appreciated that some or all ofthe techniques described herein may be applied in an MR environment ormore generally in other XR environments, and in VR environments.

As another example, embodiments are described in connection withdevices, such as wearable devices. It should be appreciated that some orall of the techniques described herein may be implemented via networks(such as cloud), discrete applications, and/or any suitable combinationsof devices, networks, and discrete applications.

Such alterations, modifications, and improvements are intended to bepart of this disclosure, and are intended to be within the spirit andscope of the disclosure. Further, though advantages of the presentdisclosure are indicated, it should be appreciated that not everyembodiment of the disclosure will include every described advantage.Some embodiments may not implement any features described asadvantageous herein and in some instances. Accordingly, the foregoingdescription and drawings are by way of example only.

The above-described embodiments of the present disclosure can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. Such processorsmay be implemented as integrated circuits, with one or more processorsin an integrated circuit component, including commercially availableintegrated circuit components known in the art by names such as CPUchips, GPU chips, microprocessor, microcontroller, or co-processor. Insome embodiments, a processor may be implemented in custom circuitry,such as an ASIC, or semi-custom circuitry resulting from configuring aprogrammable logic device. As yet a further alternative, a processor maybe a portion of a larger circuit or semiconductor device, whethercommercially available, semi-custom or custom. As a specific example,some commercially available microprocessors have multiple cores suchthat one or a subset of those cores may constitute a processor. Though,a processor may be implemented using circuitry in any suitable format.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.In the embodiment illustrated, the input/output devices are illustratedas physically separate from the computing device. In some embodiments,however, the input and/or output devices may be physically integratedinto the same unit as the processor or other elements of the computingdevice. For example, a keyboard might be implemented as a soft keyboardon a touch screen. In some embodiments, the input/output devices may beentirely disconnected from the computing device, and functionallyintegrated through a wireless connection.

Such computers may be interconnected by one or more networks in anysuitable form, including as a local area network or a wide area network,such as an enterprise network or the Internet. Such networks may bebased on any suitable technology and may operate according to anysuitable protocol and may include wireless networks, wired networks orfiber optic networks.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, the disclosure may be embodied as a computer readablestorage medium (or multiple computer readable media) (e.g., a computermemory, one or more floppy discs, compact discs (CD), optical discs,digital video disks (DVD), magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other tangible computer storage medium) encoded with one ormore programs that, when executed on one or more computers or otherprocessors, perform methods that implement the various embodiments ofthe disclosure discussed above. As is apparent from the foregoingexamples, a computer readable storage medium may retain information fora sufficient time to provide computer-executable instructions in anon-transitory form. Such a computer readable storage medium or mediacan be transportable, such that the program or programs stored thereoncan be loaded onto one or more different computers or other processorsto implement various aspects of the present disclosure as discussedabove. As used herein, the term “computer-readable storage medium”encompasses only a computer-readable medium that can be considered to bea manufacture (i.e., article of manufacture) or a machine. In someembodiments, the disclosure may be embodied as a computer readablemedium other than a computer-readable storage medium, such as apropagating signal.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of the present disclosure asdiscussed above. Additionally, it should be appreciated that accordingto one aspect of this embodiment, one or more computer programs thatwhen executed perform methods of the present disclosure need not resideon a single computer or processor, but may be distributed in a modularfashion amongst a number of different computers or processors toimplement various aspects of the present disclosure.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconveys relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Various aspects of the present disclosure may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Also, the disclosure may be embodied as a method, of which an examplehas been provided. The acts performed as part of the method may beordered in any suitable way. Accordingly, embodiments may be constructedin which acts are performed in an order different than illustrated,which may include performing some acts simultaneously, even though shownas sequential acts in illustrative embodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

What is claimed is:
 1. A portable electronic system comprising: at leastone sensor configured to capture three-dimensional (3D) informationabout objects in a physical world; local memory storing computerexecutable instructions; a transceiver configured for communication overa computer network with remote memory; and a processor communicativelycoupled to the at least one sensor, the local memory, and thetransceiver, wherein the processor executes the computer executableinstructions to provide a 3D representation of a portion of the physicalworld based at least in part on the 3D information about the objects inthe physical world, wherein: the 3D representation of the portion of thephysical world comprises a plurality of blocks, each block includingvalues representing objects in a region of the portion of the physicalworld at a corresponding point in time and indicating a floor; and thecomputer executable instructions comprise instructions for: identifyinga subset of the plurality of blocks corresponding to incomingsensor-captured information, generating a new version of a block of thesubset of the plurality of blocks based on a comparison between theincoming sensor-captured information and the subset of the plurality ofblocks, and rendering a virtual object based on the new version of theblock and its adjacent blocks of the subset.
 2. The portable electronicsystem of claim 1, wherein the values indicating the floors of theblocks are determined based at least in part on the 3D information aboutobjects in the physical world.
 3. The portable electronic system ofclaim 2, wherein the at least one sensor comprises image sensors.
 4. Theportable electronic system of claim 1, wherein at least a portion of theat least one sensor is configured to provide at least a portion of theincoming sensor-captured information that a location of the portableelectronic system is determined based at least in part on.
 5. Theportable electronic system of claim 4, wherein the location of theportable electronic system is of varying degrees of granularity.
 6. Theportable electronic system of claim 4, wherein the location of theportable electronic system comprises the values indicating the floors ofcorresponding blocks.
 7. The portable electronic system of claim 1further comprising: one or more of magnetometers, altimeters, and GPSsensors configured to provide at least a portion of the incomingsensor-captured information.
 8. The portable electronic system of claim7, wherein the values indicating the floors of the blocks are determinedbased at least in part on data captured by the one or more ofmagnetometers, altimeters, and GPS sensors.
 9. The portable electronicsystem of claim 7, wherein a location of the portable electronic systemis determined based at least in part on data captured by the one or moreof magnetometers, altimeters, and GPS sensors.
 10. The portableelectronic system of claim 9, wherein the location of the portableelectronic system is of varying degrees of granularity.
 11. A method ofoperating a portable electronic system, the method comprising: with atleast one sensor, capturing sensor information about a physical world;and with at least one processor: processing the sensor information to:produce a representation of a portion of the physical world proximatethe portable electronic device, the representation of the portion of thephysical world comprising a plurality of blocks, each block includingvalues representing objects in a region of the portion of the physicalworld at a corresponding point in time; associate altitude informationwith the portion of the physical world; generate a new version of ablock of the plurality of blocks based on a comparison between incomingsensor-captured information and the plurality of blocks; and render avirtual object based on the new version of the block and its adjacentblocks of the plurality of blocks.
 12. The method of claim 11, whereinthe altitude information comprises an identifier of a floor of abuilding.
 13. The method of claim 12, wherein associating altitudeinformation with the portion of the physical world comprises processingan output of a visual sensor to detect an indicator of the floor. 14.The method of claim 12, wherein associating altitude information withthe portion of the physical world comprises tracking changes in floor.15. The method of claim 12, wherein tracking changes in floor comprisesprocessing visual images to detect traversal of a stairway or escalator.16. The method of claim 11, wherein the representation of the portion ofthe physical world is a sparse representation.
 17. The method of claim16, further comprising: sending a localization request (a) based on therepresentation of the portion of the physical world; and (b) includingthe altitude information.
 18. The method of claim 11, wherein therepresentation of the portion of the physical world is a first portionof a dense representation.
 19. The method of claim 18, wherein: thedense representation comprises a plurality of portions, including thefirst portion, each portion comprising associated altitude information;and the method further comprises performing occlusion processing basedon the plurality of portions, the occlusion processing comprisingordering surface information in portions of the plurality of portionsbased on the altitude information associated with the respectiveportions.
 20. A non-transitory computer-readable medium withinstructions stored thereon, that when executed on a processor, performthe acts comprising: processing sensor information about a physicalworld to: produce a representation of a portion of the physical worldproximate a portable electronic device, the representation of theportion of the physical world comprising a plurality of blocks, eachblock including values representing objects in a region of the portionof the physical world at a corresponding point in time; associatealtitude information with the portion of the physical world; generate anew version of a block of the plurality of blocks based on a comparisonbetween incoming sensor information and the plurality of blocks; andrender a virtual object based on the new version of the block and itsadjacent blocks of the plurality of blocks.